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1  Oth  Annual  Systems  Engineering  Conference 
“Focusing  on  Improving  Performance  of  Defense  Systems  Programs ” 

San  Diego,  CA 
22-25  October  2007 
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Agenda 


Tuesday.  23  October  2007 

Keynote  Addresses: 

•  Hon  James  Finley.  Deputy  Under  Secretary  of  Defense,  Acquisition  and  Technology 

•  Hon  Charles  McOueary.  Director,  Operational  Test  and  Evaluation 

Plenary  Session:  Executive  Panel: 

•  Mr.  Mark  Schaeffer.  Director,  Office  of  Under  Secretary  of  Defense  of  Acquisition,  Technology  and  Logistics;  Director,  Systems  and  Software  Engineering 

Panelists: 

•  Mr.  Terry  Jaggers .  SAF/AQR  -  Science,  Technology,  and  Engineering 

•  Mr.  Carl  Siel.  Chief  Engineer,  Office  of  the  Assistant  Secretary  of  the  Navy  Research,  Development  and  Acquisition 

•  Mr.  Doug  Wiltsie.  HODA.  OASA  (ALT) 


Track  1 

•  “Update:  OSD  Systems  Engineering  Revitalization  Efforts,”  Col  Richard  Hoeferkamp,  USAF 

•  “The  Effectiveness  of  Systems  Engineering:  On  Federal  (DoD)  System  Development  Programs”,  Mr.  A1  Mink,  SRA  International 

•  “Tools  and  Resources  to  Enable  Systems  Engineering  Improvement,”  Mr.  Michael  Kutch,  SPAWAR  Systems  Center  Charleston 

•  “Sound  Systems  Engineering  Assures  Proper/Early  Producibility”,  Dr.  Thomas  Christian,  Aeronautical  Systems  Center 

•  “Realization  of  Systems  Engineering  For  the  Future”,  Ms.  Karen  Bausman,  AF  Center  for  Systems  Engineering 

Track  2 

•  “Developmental  Test  &  Evaluation  Policy  Vectors”,  Ms.  Darlene  Mosser-Kemer  OUSD  (AT&L) 

•  “Test  Strategy  Done  Early  Drives  Test  Planning  and  Successful  Testing”,  Mr.  William  Lyders,  AS  SETT,  Inc. 

•  “Applying  Design  of  Experiments  Methodology  to  Sortie  Generation  Rate  Evaluation”,  Mr.  Joseph  Tribble, _AVW_“Implementing  a  Systems  Engineering 
Risk  Program  in  a  Sustainment  Environment”,  Mr.  James  Miller,  USAF 

•  “Joint  Safety  Review  Process  Study”,  Ms.  Paige  Ripini,  Booz  Allen  Hamilton 

Track  3 

•  “DoD  Systemic  Root  Cause  Analysis”,  Mr.  Dave  Castellano,  OUSD  (AT&L) 

•  “Applying  Systems  Engineering  During  Pre-Acquisition  Activities”,  Lt  Col  Mark  Wilson,  USAF 

•  “Reforming  the  DoD  Acquisition  Process — A  Systems  Engineering  Approach”,  Mr.  Stephan  Ward,  U.S.  Air  Force 

•  “The  Effectiveness  of  Systems  Engineering:  On  Federal  System  Development  Programs”,  Mr.  Alan  Mink,  SRA  International 

Track  4 

•  “Environment,  Safety,  and  Occupational  Health  (ESOH) — Design  Considerations  to  Support  Sustainability  and  Readiness”,  Ms.  Patricia  Huheey,  ODUSD 
(I&E) 

•  “Real-Time  Diagnostics  for  High  Availability  Systems”,  Mr.  Edward  Beck,  Computer  Sciences  Corporation 

•  “Sparing  Satellites  Comparative  Strategies  of  On-orbit  &  In-factory  Storage”,  Mr.  James  Mazzei,  The  Aerospace  Corporation 

Track  5 

•  “Acquisition  M&S  Master  Plan  Implementation  Status”,  Mr.  Michael  Truelove,  SAIC 

•  “Establish  M&S-Related  Guidelines  for  Solicitations,  Source  Selections,  and  Contracting”,  Mr.  Michael  Truelove,  SAIC 

•  “Modeling  and  Simulation  Resource  Reuse  Business  Model”,  Mr.  Dennis  Shea,  Center  for  Naval  Analyses 
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•  “Modeling  and  Simulation  Support  Plan”,  Mr.  David  Henry,  Lockheed  Martin 

•  “Modeling  and  Simulation  Education  for  the  Acquisition/T&E  Workforce:  Requirements  Analysis”,  Mr.  David  Olwell,  NPS 
Track  6 

•  “The  Use  of  Enterprise  Architecture  to  Support  the  Development  of  the  Next  Generation  Air  Transportation  System  (NextGen)”,  Mr.  David  Synder,  The 
MITRE  Corp. 

•  “Complex  Systems  of  Systems:  The  Dual  Challenge”,  Mr.  Phillip  Boxer,  Software  Engineering  Institute 

•  “Systems  Engineering  in  the  Cognitive  and  Social  Domains  of  Net  Centric  Operations”,  Dr.  Abe  Meilich,  Lockheed  Martin 


Track  7 

•  “Simplifying  &  Scaling  Engineering  Processes:  Unifying  Business  Units  and  Engineering  Disciplines”,  Mr.  Raymond  Jorgensen,  Rockwell  Collins 

•  “NAVAIR  Systems  Engineering  Revitalization”,  Mr.  Michael  Gaydar,  Department  of  Navy,  NAVAIR 

•  “Integrating  Engineering  Project  Management  and  Product  Development  Processes”,  Mr.  Raymond  Jorgensen,  Rockwell  Collins 

•  “Engineering  for  System  Assurance — Legacy,  Life  Cycle,  Leadership”,  Mr.  Paul  Croll,  Computer  Sciences  Corporation 

Track  8 

o  “DoD  Software  Engineering  and  System  Assurance”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  Systems  and  Software  Engineering 
o  “The  Integrated  Software  and  Systems  Engineering  Curriculum  Project:  Creating  a  Reference  Curriculum  for  Graduate  Software  Engineering 
Education”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  Systems  and  Software 
o  “Requirements  for  a  Chief  Software  Engineer  in  a  DoD  Acquisition  Agency”,  Mr.  A1  Florence,  The  MITRE  Corporation 
o  “Developing  an  Integrated  Process  Methodology  for  Interim  Software  Releases”,  Mr.  Tim  Woods,  Southern  Methodist  University 

Wednesday.  24  October  2007 
Track  1 

o  “Change  Management  of  UML-Based  Systems  Engineering  Artefacts”,  Mr.  David  Price,  Eurostep 
o  “A  Day  in  the  Life  of  a  Verification  Requirement”,  Mr.  Stephen  Scukanec,  Northrop  Grumman  Corporation 
o  “How  to  Measurably  Improve  Your  Requirements”,  Mr.  Timothy  Olson,  Lean,  Solutions  Institute,  Inc. 
o  “Case  Studies:  A  Common  Language  Between  Engineers  and  Managers”,  Capt  DeWitt  Latimer,  USAF 
o  “A  Strategy  for  Improved  System  Assurance”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 

°  “Discussion  of  the  U.S.  Army  RDECOM  APS  Objective  Trade  Study”,  Mr.  Frank  Salvatore,  High  Performance  Technologies,  Inc. 
o  “Program  Support  Review  Deep  Dive”,  Mr.  Peter  Nolte,  OUSD/SSE 

Track  2 

o  “An  Update  on  the  DT&E  Committee’s  Recommended  Policy  Changes  to  DoD  5000”,  Col  Richard  Stuckey,USAF,OUSD  (AT&L)/SSE/  DT&E 
o  “System  Test  and  Evaluation  in  the  DARPA  Immune  Building  Demonstration  Program”,  Mr.  Mark  Saxon,Battelle 

o  “  Modeling  and  Simulation  in  the  Navy  Warfare  Systems  Test  &  Evaluation  Enterprise”,  Ms.  Shala  Malone, Navy  Program  Executive  Office  Integrated 
Warfare 

o  “Joint  Mission  Environment  Test  Capability  (JMETC)”,  Mr.  Richard  Lockhart,  Test  Resource  Management  Center 
o  “Testing  Concept  of  Operations  in  DoD’s  Net  Centric  Environment”,  Mr.  Steve  Reeder,  South  Carolina 
o  Research  Authority  (SCRA 

o  “Do  it  right,  do  it  early;  Do  it  early,  do  it  right” — Considerations  for  the  Early  Stages  of  Concept,  System,  and  Systems  of-Systems  Definition”,  Mr.  Jeff 
Loren,  MTC  Technologies,  Inc.  (SAF/AQRE) 

o  “Applications  of  Systems  Engineering  to  Pre-Milestone  A  Projects”,  Ms.  Lori  Zipes, Naval  Surface  Warfare  Center  PC 
o  “Systems  Engineering  in  a  Systems  of  Systems  Environment  -  Defense  Update”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 
o  “ASW  System-of- Systems  Engineering  Analysis  Applied  in  an  LCS  ASW  Integration  Pilot  Project”,  Mr.  G.  Richard  Thompson,  JHU/APL 


Track  3 

o  “Systems  Engineering  Plan  Preparation  Guide  Update”,  Mr.  Chester  Bracuto,  OSD/AT&L/A&T/SSE/ED 
o  “  Toward  a  Unified  Systems  Engineering  Plan”,  Mr.  Robert  Scheurer,  Boeing  Integrated  Defense  Systems 
o  “Integrating  Risk  &  Knowledge  Management”,  Mr.  David  Lengyel,  NASA 

o  “Systems  Engineering  and  Program  Management — How  Different  are  They?”,  Ms.  Lori  Zipes,  Naval  Surface  Warfare  Center  PC 
o  “Systems  Engineering  Analysis  to  Improve  Concept  Development  of  Complex  Defense  Systems”,  Mr.  Michael  Harper,  SPAWAR  Systems  Center 
Charleston 

o  “  The  Joint  Partnership  Between  Program  Management  and  Systems  Engineering”,  Mr.  Samuel  Son,  The  Boeing  Company 
o  “Lockheed  Martin  Aeronautics  Company  Approach  to  Solving  Systemic  Development  Program  Issues”,  Mr.  John  Weaver,  Lockheed  Martin 
Aeronautics  Company 

o  “Airborne  Signals  Intelligence  Payload  (ASIP)  Program  Integrated  Risk  Management”,  Mr.  Danial  Bolek,  USAF 
o  “Improvements  to  the  Risk  Management  Process”,  Mr.  Doug  Atkinson,  USAF 
o  “Integrated  Risk  and  Earned  Value  Management”,  Mr.  Paul  Davis,  Northrop  Grumman 

o  “Application  of  Risk  Management  Practices  to  NNSA  Tritium  Readiness  Subprogram”, Mr.  Sham  Shete,  Washington  Savannah  River  Co. 

Track  4 

o  “Defining  Lean  Service  and  Maintenance  Processes”,  Mr.  Timothy  Olson,  Lean  Solutions  Institute,  Inc. 

o  “Modeling  Integrated  Logistics  Systems  to  Support  Transformation  in  the  U.S.  Coast  Guard”,  Mr.  Patrick  Cumby,  VectorCSP,  L 
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“Asset-Based  PBL  for  Navy  Warships  -  A  Case  Study  for  LCS  Class  Ships”,  Mr.  Michael  Mahon,  Lockheed  Martin 
o  “Integrated  Diagnostics  Closed  Loop  Data  System  (At  the  Point  of  Use)  (Support  Systems  Knowledge  Engineering  Enhances  Traditional  Support 
Equipment  Systems  Engineering),”  Mr.  Steven  Head,  Boeing 
o  “Aging  Aircraft  Sustainment  with  Non-Standard  Engineering”,  Ms.  Kendal  Hinton,  Georgia  Tech  Research  Institute 
o  “Maintaining  System  Viability  for  the  Long  Term”,  Mr.  Peter  Henry,  BAE  Systems  Land  and  Armament 
o  “C-17  Program  Applies  Systems  Engineering  to  a  Large  Improvement  Project”,  Mr.  Brent  Theodore, The  Boeing  Company 

Track  5 

o  “ Advancing  the  FEDEP  for  Simulation  Based  Acquisition ”,  Dr.  Katherine  Morse,  SAIC 

o  “Acquisition  M&S  Community  Sponsored  M&S  Project:  Standardized  Documentation  for  Verification,  Validation,  and  Accreditation”,  Mr.  Kevin 
Charlow,  (paper)  (slides)  Space  and  Naval  Warfare  Systems  Center  Charleston 
o  “A  Methodology  for  Assessing  &  Prioritizing  the  Risks  Associated  with  the  Level  of  Verification,  Validation  and  Accreditation  (VV&A)  of  Models 
and  Simulations”,  Dr.  James  Elele,  U.S.  Navy 

o  “Experiences  in  Applying  SysML  to  Develop  Interoperable  Torpedo  Modeling  and  Simulation  Components”,  Mr.  Thomas  Haley,  Naval  Undersea 
Warfare  Center 

o  “Unifying  Systems  Engineering  Simulations”,  Mr.  Ryan  O’ Grady,  Cybernet  Systems  Corporation 
o  “Information  Modeling  for  Systems  Integration”,  Ms.  Claudia  Rose,BBII 

o  “Simulation  Supported  Decision  Making”,  (slidesl)  (slides  2)  Mr.  Gene  Allen,  MSC  Software  Corporation 
Track  6 

o  “Achieving  Agility  in  Cyberspace”,  Mr.  Phillip  Boxer,  Software  Engineering  Institute 
o  “Application  of  Autonomic  Agents  for  Global  Information  Grid”,  Mr.  David  Cox,  University  of  Arizona 
o  “Architecture-Based  Concept  Evaluation  in  Support  of  JCIDS”,  Dr.  David  Jacques,  Air  Force  Institute  of  Technology 
o  “System  of  Systems  Implications  for  Operational  Test”,  Dr.  John  Colombi,  Air  Force  Institute  of  Technology 
°  “Case  Study:  Net  Centric  Mission  Thread  Modeling  and  Analysis”,  Dr.  Prem  Jain,  MITRE 

o  “Quantitative  Comparison  of  Alternative  Designs  for  a  JC3M  System”,  Mr.  Gregory  Miller,  Naval  Postgraduate  School 
o  “Advanced  Net  Centric  Simulation  for  Aerospace  Command  and  Control”,  Ms.  Kimberly  Kendall,  753d  ELSG/NEM,  ESC,  USAF 


Track  7 

o  “CMMI— Next  Steps”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 

o  “CMMI  Instructional  Challenges  to  Systems  Engineers  in  Small  Settings”,  Dr.  Mary  Anne  Herndon,  Transdyne  Corporation 
“FISMA  Operational  Controls  and  Their  Relationship  to  Process  Maturity”,  Ms.  Rhonda  Henning,  Harris  Corporation 

o  “Executing  a  Successful  CMMI  Maturity  Level  3  SCAMPI  For  SPAWAR  Systems  Center  Charleston”,  Mr.  Michael  Kutch,  SPAWAR  Systems  Center 
Charleston 

o  “CMMI  for  Services:  Re-Introducing  the  CMMI  for  Services  Constellation”,  Mr.  Craig  Hollenbach,  Northrop  Grumman  Corporation 

o  “How  to  Paint  a  Room:  The  Role  of  Specs  &  Standards  in  SE”,  Mr.  Robert  Kuhnen,  USAF 

o  “Continuous  Improvement  at  the  Organization,  Team,  and  Individual  Levels  — Lessons  Learned  Integrating  CMM,TSP,  and  PSP  and  Why  All  Three 
are  Needed”,  Mr.  Girish  Seshagiri, Advanced  Information  Services,  Inc 

o  “Addressing  Environment,  Safety  and  Occupational  Health  Issues  for  the  Mine  Resistant  Ambush  Protected  (MRAP)  Vehicle  Program”,  Ms.  Jennifer 
Malone,  EG&G 

o  “Anatomy  of  an  Award  Winning  Safety  Program:  A  Case  Study  of  the  SSGN  OHIO  Class  Conversion  Safety  Program”,  Mr.  Ricky  Milnarik,  Electric 
Boat  Corporation 

o  “The  Safety  of  Unmanned  Systems:  The  Process  Used  to  Develop  Safety  Precepts  for  Unmanned  Systems”,  Mr.  Mike  Demmick,  NOSSA 
Track  8 


■  “Defining  Software  Component  Specifications:  An  Open  Approach”,  Mr.  Kenneth  Klein, Computer  Sciences  Corporation 

■  “System  Engineering  and  Software  Exception  Handling  (SEH)”,  Mr.  Herbert  Hecht,  SoHaR  Incorporated 

■  “A  Convergence  of  Technologies  for  Better  Software  NOW!”,  Ms.  Dorothy  Acton,  Lockheed  Martin  IS&GS 

■  “Identifying  Acquisition  Patterns  of  Failure  Using  System  Archetypes”,  Mr.  William  Novak,  Software  Engineering  Institute 

■  “Revitalizing  Education  and  Training  in  Systems  Engineering” ,  Dr.  Don  Gelosh,  Department  of  Defense,  OSD(AT&L)/SSE/ED 

■  “Customer-Driven,  Partnership-Based  Systems  Engineering  Education  and  Training”,  Mr.  Jerrell  Stracener,  Southern  Methodist  University 

■  “Application  of  Systems  Engineering  Principles  in  the  Design  of  Acquisition  Workforce  Curricula”,  (paper)  (slidel  Dr.  David  Olwell.  Naval 
Postgraduate  School 


Thursday.  25  October  2007 
Track  1 


■  “USAF  Type  Certification  of  Commercial  Derivative  Aircraft”,  Mr.  Thomas  Morgan,  USAF 

■  “Global  Positioning  System  Case  Study”,  Mr.  Randall  Bullard,  Air  Force  Center  for  Systems  Engineering 

Track  2 

■  “Implementing  the  Technology  Maturity  Vector”,  Mr.  Joseph  Terlizzese,  Systems  Engineering  Support  Office 

■  “Technology  Readiness  Assessments;  Milestone  B  Certification  Requirement  for  Technologies  to  be  Demonstrated  in  a  Relevant  Environment”, 
Dr.  Jay  Mandelbaum,  Institute  for  Defense  Analyses 

■  “Meeting  Enterprise  System  Engineering  Challenges  for  the  U.S.  Next  Generation  Air  Transportation  System  (NextGen)”,  Mr.  Jerry  Friedman, 
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The  MITRE  Corporation 

■  “Sensor  Resource  Allocation  as  a  Driver  in  System  Concept  Development”,  Mr.  Ravi  Moorthy,  Lockheed  Martin  MS2 
Track  3 

■  “Managing  Requirements  to  Manage  Scope  in  the  Case  of  MUOS”.  Ms.  Christy  Howard.  Maxim  Systems,  Inc. 

■  “Organizational  Leadership  and  Management  Dynamics  for  Technical  Execution  in  Acquisition  Programs”.  Mr.  Francis  Sisti.  Aerospace 
Corporation 

■  “C-17  Systems  Engineering  Process  to  Prioritize  Material  Improvement  Program  (MIP)  Projects”,  Mr.  Thomas  Condron,  USAF  (516 
AESG/ASC) 

■  “How  to  Talk  to  a  Program  Manager”.  Dr.  John  Mishler.  Software  Engineering  Institute 

■  “U.S.  Department  of  Defense  (DoD)  Approach  to  Best  Practices:  Building  Evidence  for  Practice  Selection  Based  on  Real  Experiences”,  Dr. 
Forrest  Shull,  Fraunhofer  Center  Maryland 

Track  4 

■  “Strategic  Focus:  Reduction  of  Total  Ownership  Costs  (R-TOC)  and  Value  Engineering  (VE)”,  Dr.  Danny  Reed,  Institute  for  Defense  Analyses 

■  “Progress  Toward  an  Empirical  Relationships  Between  Reliability  Investments  and  Life-Cycle  Support  Cost”,  Dr.  James  Forbes,  LMI 

■  “Innovation  Strategies  for  Affordable  Readiness”.  Mr.  Tom  Choinski.  Naval  Undersea  Warfare  Center 

■  “Implementing  a  Systems  Engineering  Risk  Program  in  a  Sustainment  Environment”,  Mr.  James  Miller,  USAF 

■  “Asset-Based  PBL  for  Navy  Warships  -  A  case  study  for  LCA  Class  Ships”,  Mike  Mahon 

■  “The  Deployment  Readiness  Service:  The  Challenges  of  Implementing  a  Service  Oriented  Architecture  in  a  Legacy  System  Environment”,  Mr. 
George  Dalton,  USAF 

Track  5 

■  “Aircraft  Flight  Simulator  Acceptance  Criteria”,  Mr.  Dean  Carico,  NAWCAD  PAX 

■  “Computer  Modeling  to  Solve  Problems  with  the  T-38  Propulsion  Modernization  Program”,  Mr.  Randall  Wimer,  USAF  -  FVB 

■  “Efficacy  of  Modeling  &  Simulation  in  Defense  Life  Cycle  Engineering”,  Mr.  Donald  Cox,  Raytheon  Missile  Systems 

■  “Modeling  Integrated  Logistics  Systems  to  Support  Transformation  in  the  U.S.  Coast  Guard”.  Mr.  Patrick  Cumbv.  Vector  CSP,  LLC 

■  “Generic  Sensor  Model”.  Dr.  Stanley  Hack.  Lockheed  Martin  MS-2 

■  “Event  Timeline  Analysis  in  Multi  Mission  Scenarios  with  System  Simulation  Models”,  Mr.  Ravi  Moorthy,  Lockheed  Martin  MS2 
Track  6 

■  “Testing  Concept  of  Operations  in  DoD’s  Net  Centric  Environment”,  Mr.  Steve  Reeder,  South  Carolina  Research  Authority  (SCRA) 

■  “Agile  Governance  for  SOA-Based  Military  Systems  of  Systems”,  Mr.  Robert  Beck,  Villanova  University 

■  “Reducing  Acquisition  Costs  Through  Incremental  Upgrades  by  Migrating  to  SOA”,  Mr.  Tim  Greer,  Lockheed  Martin  Corporation 
Track  7 

■  “The  DoD’s  Proactive  Approach  to  Emerging  Contaminants:  Managing  Risks  Today  for  Tomorrow’s  Warfighter  and  Mission  Readiness”,  Dr. 
Carole  LeBlanc,  Office  of  the  Deputy  Under  Secretary  of  Defense 

■  “ Safe-Escape  Analysis  System  Sa  fety  Engineering  Study”,  (paper!  (slide)  Mr.  David  Hall,  SURVICE  Engineering  Company 

Track  8 


■  “Development  of  Systems  Engineers  in  the  Sensors  &  SONAR  Systems  Department”.  Mr.  Lawrence  Lazar.Naval  Undersea  Warfare 
Center 

■  ^Systems  Engineering  and  the  Art  of  Seeing”,  Dr.  Robert  Monson,  Lockheed  Martin  Corporation 

■  “Understanding  Social  Networks- A  Key  Requirement  for  System  Engineers”,  Mr.  Karl  Selke,  Systems  Engineering  Analyst,  Evidence 
Based  Research,  Inc. 
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Conference  Agenda  {At  A  Glance} 

Sunday,  October  21, 2007 

5:00  pm  -  7:00  pm  Registration  for  Tutorials  and  General  Conference 

(Tutorials  are  an  additional  $225.00  registration  fee) 

Monday,  October  22, 2007 


7:00  am  -  6:00  pm 
7:00  am  -  8:00  am 

8:00  am  - 11:45  am 

12:00  pm  - 1:00  pm 
1:00  pm  -  5:00  pm 
5:00  pm  -  6:00  pm 


Registration 

Continental  Breakfast  for  Tutorial  Attendees  ONLY 
(Tutorials  are  an  additional  $225.00  registration  fee) 

Tutorial  Tracks 

(Please  refer  to  pages  4-5  for  Tutorial  schedule) 

Lunch  for  Tutorial  Attendees  ONLY 
Tutorial  Tracks  Continued 

Reception  in  the  Regatta  Pavilion  (Open  to  All  Participants) 


Tuesday,  October  23, 2007 


7:15  am -6:30  pm 
7:15  am  -  8:15  am 


Registration 
Continental  Breakfast 


8:15  am  -  8:30  am 


8:30  am  -  9:45  am 


9:45  am  - 10:00  am 


Introductions  8C  Opening  Remarks: 

Mr.  Sam  Campagna,  Director ;  Operations ,  NDIA 
Mr.  Boh  Rassa,  Director ;  Systems  Supportability,  Raytheon , 

Chain  Systems  Engineering  Division,  NDIA 

Keynote  Addresses: 

HON James  Finley ,  Deputy  Under  Secretary  of  Defense,  Acquisiton  &  Technology 
HON  Charles  McQueary,  Director,  Operational  Test  &  Evaluation 


Break 


2  10th  Annual  Systems  Engineering  Conference  ♦  October  22-25,  2007  ♦  San  Diego,  CA 


10:00  am  - 12:00  pm 

Plenary  Session:  Executive  Panel 

Moderator: 

Mn  Mark  Schaeffer ;  Director ;  Office  of  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics;  Director ;  Systems  and  Software  Engineering 

Panelists: 

Mn  Terry  Jaggers,  SAF/AQR  -  Science,  Technology,  and  Engineering 

Mn  Carl  Siel,  Chief  Engineer,  Office  of  the  Assistant  Secretary  of  the  Navy 
Research,  Development  and  Acquisition 

Mn  Doug  Wiltsie,  HQDA ,  OASA  (ALT) 

12:00  pm  - 1:30  pm 

Luncheon  with  Speaker  in  the  Regatta  Pavilion 

Mn  Mike  Kern,  Senior  Systems  Engineer,  OASD  (Nil) 

1:30  pm  -  5:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

5:00  pm  -  6:30  pm 

Reception  in  the  Regatta  Pavilion 

Wednesday,  October  24, 2007 

7:00  am  -  5:00  pm  Registration 


7:00  am  -  8:00  am 

Continental  Breakfast 

8:00  am  - 12:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

12:00  pm  - 1:30  pm 

Awards  Luncheon  in  the  Regatta  Pavilion 

1:30  pm  -  5:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

Thursday,  October  25, 2007 

7:00  am  -  3:00  pm  Registration 


7:00  am  -  8:00  am 

Continental  Breakfast 

8:00  am  - 12:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

12:00  pm  - 1:00  pm 

Luncheon  in  the  Regatta  Pavilion 

1:00  pm  -  3:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

3:00  pm 

Conference  Adjourns 

7:00  am  Registration  &  Continental  Breakfast 
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Tutorial  Sessions  -  Monday,  October  22, 2007 

8:00  am  -  9:45  am  10:15  am  - 1 1 :45  am 


Track  1 

Tutorial 

Session  1A1 

5305  -  Are  we  Ready  for  CMMI®?  If  not. 
Lets  Fix  Ourselves 

Mr.  Al  Florence ,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1A2 

5454  -  Cost  As  an  Independent  Variable 
and  Trade  Studies 

Mr,  Ed  Casy,  Raytheon  Missile  Systems 

Track  3 

Tutorial 

Session  1A3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices 

Mr.  Gary  Langford,  The  Naval  Postgraduate 

School 

Track  4 

Tutorial 

Session  1A4 

5498  -  System  Verification  Organization 

Mr,  Jeffrey  Grady,  JOG  System  Engineering, 
Inc. 

Track  5 

Tutorial 

Session  1A5 

Track  6 

Tutorial 

5542  -  Best-In-Class  Early  Defect 

Detection  and  Defect  Prevention 

Session  1A6 

Mr.  Timothy  Olson,  Lean  Solutions  Institute, 
Inc. 

Track  7 

Tutorial 

5784  -  Operational  Concepts:  Using  Cases 
&  Scenarios  to  Understand  Users  Needs 

Session  1A7 

Mr.  Raymond  Jorgensen,  Rockwell  Collins 

Track  8 

Tutorial 

Session  1A8 

5577  -  Introduction  to  SysML  & 

Object  Oriented  Systems  Engineering 
Methodology  (OOSEM) 

Mr.  Ahe  Meilich,  Lockheed  Martin 

Track  1 

Tutorial 

Session  1B1 

5305  -  Are  we  Ready  for  CMMI®?  If  not. 
Lets  Fix  Ourselves  (Cont'd) 

Mr.  Al  Florence,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1B2 

5454  -  Cost  As  an  Independent  Variable  and 
Trade  Studies  (Contd) 

Mr.  Ed  Casy,  Raytheon  Missile  Systems 

Track  3 

Tutorial 

Session  1B3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices  (Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate 
School 

Track  4 

Tutorial 

Session  1B4 

5498  -  System  Verification  Organization 
(Cont'd) 

Mr.  Jeffrey  Grady,  JOG  System  Engineering, 
Inc. 

Track  5 

Tutorial 

Session  1B5 

Track  6 

Tutorial 

Session  1B6 

5542  -  Best-In-Class  Early  Defect 

Detection  and  Defect  Prevention  (Cont'd) 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc. 

Track  7 

Tutorial 

Session  1B7 

5784  -  Operational  Concepts:  Using  Cases 
&  Scenarios  to  Understand  User's  Needs 
(Cont'd) 

Mr.  Raymond  Jorgensen,  Rockwell  Collins 

Track  8 

Tutorial 

Session  1B8 

5577  -  Introduction  to  SysML  & 

Object  Oriented  Systems  Engineering 
Methodology  (OOSEM)  (Cont'd) 

Mr.  Ahe  Meilich,  Lockheed  Martin 
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Buffet  Lunch 


3:15  pm  -  5:00  pm 


1 :00  pm  -  2:45  pm 


Track  1 

Tutorial 

Session  1C1 

5307  -  Requirements  Development  and 
Management 

Mr.  Al  Florence ,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1C2 

5326  -  Integrating  Systems  Engineering 
with  Earned  Value  Management 

Mr.  Paul  Solomon,  Performance-Based  Earned 
Value • 

Track  3 

Tutorial 

Session  1C3 

5404  -  How  to  Reduce  Schedule  Uncertainty 
by  Integrating  Sound  Management  Methods 
with  Systems  Engineering  Best  Practices 
(Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate  School 

Track  4 

Tutorial 

Session  1C4 

5511  -  Modeling  Sustainment  and  Risk 
Mitigation  for  Net-Enabled  Realities 

Mr.  Philip  Boxer,  Software  Engineering 
Institute/CMU 

Track  5 

Tutorial 

Session  1C5 

5540  -  Introduction  to  Reliability  Analysis 

Dr.  Meng-Lai  Yin,  Raytheon  Company 

Track  6 

Tutorial 

Session  1C6 

5544  -  How  to  Define  Practical  Systems 
Engineering  Metrics 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc. 

Track  7 

Tutorial 

Session  1C7 

5582  -  Leading  Effective  System  of 

Systems  (SoS)  Technical  Reviews 

Mr.  David  Walden,  Sysnovation,  LLC 

Track  8 

Tutorial 

Session  1C8 

5577  -  Introduction  to  SysML  &  Object 
Oriented  Systems  Engineering 

Methodology  (OOSEM) 

Dr.  Ahe  Meilich,  Lockheed  Martin 

Track  1 

Tutorial 

Session  1D1 

5307  -  Requirements  Development  and 
Management  (Cont'd) 

Mr.  Al  Florence,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1D2 

5326  -  Integrating  Systems  Engineering 
with  Earned  Value  Management  (Cont'd) 

Mr.  Paul  Solomon,  Performance-Based  Earned 
Value • 

Track  3 

Tutorial 

Session  1D3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices  (Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate  School 

Track  4 

Tutorial 

Session  1D4 

5511  -  Modeling  Sustainment  and  Risk 
Mitigation  for  Net-Enabled  Realities 
(Cont'd) 

Mr.  Philip  Boxer,  Software  Engineering 
Institute/CMU 

Track  5 

Tutorial 

Session  1D5 

5540  -  Introduction  to  Reliability  Analysis 
(Cont'd) 

Dr.  Meng-Lai  Yin,  Raytheon  Company 

Track  6 

Tutorial 

Session  1D6 

5544  -  How  to  Define  Practical  Systems 
Engineering  Metrics 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc . 

Track  7 

Tutorial 

Session  1D7 

5582  -  Leading  Effective  System  of 

Systems  (SoS)  Technical  Reviews  (Cont'd) 

Mr.  David  Walden,  Sysnovation,  LLC 

Track  8 

Tutorial 

Session  1D8 

5577  -  Introduction  to  SysML  &  Object 
Oriented  Systems  Engineering 

Methodology  (OOSEM)  (Cont'd) 

Dr.  Ahe  Meilich,  Lockheed  Martin 

5:00  pm  -  6:00  pm  Reception  in  Regatta  Pavilion 


Tuesday,  October  23,  2007 


5:00  pm  -  6:00  pm  Reception  in  the  Regatta  Pavilion 


Wednesday,  October  24,  2007 


12:00  pm  Lunch  in  the  Regatta  Pavilion 


Wednesday,  October  24,  2007 
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Thursday,  October  25,  2007 

8:00  am  -  9:45  am  concurrent  sessions  10:15  am  - 12:00  pm 
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12:00  pm  Lunch  in  the  Regatta  Pavilion 


Thursday,  October  25,  2007 


concurrent  sessions 

:30  pm  -  3:00  pm 

Bayview 

A 

TRACK  2 

Systems  Engineering 
Effectiveness 

Session  4C2 

5814  -  Defense  System  and 
Large-Scale  Systems  Based  on 
Terminal  Control 

Mr.  Xiang-Wen  Xiong,  Zhongheng 
High-Tech  Institute,  Inc. 

Bayview 

B 

TRACK  4 

Logistics,  Supportability, 
and  Readiness 

Session  4C4 

5416  -  The  Deployment  Readiness 
Service:  The  Challenges  of 
Implementing  a  Service  Oriented 
Architecture  in  a  Legacy  System 
Environment 

Mr.  George  Dalton,  USAF 

yview 

C 

TRACK  5 

Modeling  & 

Simulation 

5838  -  Generic  Sensor  Model 

5852  -  Event  Timeline  Analysis 
in  Multi  Mission  Scenarios  with 
System  Simulation  Models 

5428  -  A  Prototype  Tool  for 
Concept  Design  Modeling  and 
Optimization  of  Combat  Systems 

CQ 

Session  4C5 

Dr.  Stanley  Hack,  Lockheed 

Martin  MS-2 

Mr.  Ravi  Moorthy,  Lockheed 
Martin  MS2 

Mr.  Vikram  Ganesan,  General 
Dynamics  Land  Systems 

Conference  Adjourns 


Systems  Engineering  Effectiveness: 

Mr.  A1  Brown 
Ms.  Sharon  Vannucci 
Mr.  Bob  Lyons 

Logistics  Supportability  &  Readiness: 

Mr.  Chuck  Silva 
Mr.  Joel  Moorvich 

Test  &  Evaluation  in  Systems  Engineering: 
Col  Rich  Stuckey,  USAF 
Mr.  Tom  Wissink 
Program  Management: 

Mr.  Hal  Wilson 
Modeling  &  Simulation: 

Mr.  Jim  Hollenbach 
Mr.  Gary  Belie 
Net  Centric  Operations: 

Mr.  Jack  Zavin 
Dr.  Rich  Eilers 
Dr.  Tom  Wickstrom 
Best  Practices  &  Standardization: 

Mr.  Paul  Croll 
Software: 

Dr.  Tom  Christian 
Education  &  Training: 

Mr.  George  Mooney 
Integrated  Diognostics: 

Mr.  Howard  Savage 
Mr.  Dennis  Hecht 
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1A4 

5498 
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Mr.  Jeffrey  Grady 

1A6 
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1A7 
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2C1 

5378 

DoD's  Systems  Engineering  Revitalization  Efforts — An  Update 
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Maj  Joerg  Walter,  US AF 

Dr.  Som  Soni 

2C4 
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Environment,  Safety,  and  Occupational  Health  (ESOH)  -  Design 
Considerations  to  Support  Sustainability  and  Readiness 

Ms.  Patricia  Huheey 

Ms.  Karen  Gill 

2C5 

5421 

M&S -Related  Guidelines  for  Contracting 

Mr.  Michael  Truelove 

2C5 
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Implementing  the  Acquisition  M&S  Master  Plan 
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I  THE  AEROSPACE 


CORPORATION 


•  Introduction 

•  Warm  Spare  vs.  Launch  on  Need  Comparison 

•  Analysis  of  Operational  Programs 

-  Defense  Satellite  Communications  System 

-  Tactical  Data  Relay  Satellite  System 

-  Geostationary  Operational  Environmental  Satellite 

•  Commercial  Systems 

•  Advantages  &  Disadvantages  of  OOS 

•  Conclusion 


I  THE  AEROSPACE 


CORPORATION 


UNCLASSIFIED 


introduction 


•  Time  required  to  produce  satellite 

•  Generalized  Availability  Program  (GAP) 

•  Milestone  Schedule  Elements 


| THE  AEROSPACE 

CORPORATION 


UNCLASSIFIED 


Warm  Spuire  vs*  Launch  on  Need  j 

\ 

Case  1  vs  Case  3  LON  (ESD/Aero) 


Case  1  W.S.  VS  Case  3  Len  (ESD/Aero) 


I  THE  AEROSPACE 


CORPORATION 


UNCLASSIFIED 


•  A  military  satellite  communications  system 

-  240  day  LON  =  $  1 M  recurring 

-  60  day  LON  =  $22.5  M  non-recurring  &  $5.3  M 
recurring 


•  T actical  Data  Relay  Satellite  System 

-  On-orbit  savings  =  $6.5  M  over  2  years 


| THE  AEROSPACE 

CORPORATION 


UNCLASSIFIED 


Commercial  systems  universally  store  on  orbit 
to  take  advantage  of  surge  requirements  and 
difficult  maintenance  issues. 


I  THE  AEROSPACE 


CORPORATION 


•  Advantages:  Cost  (with  assumptions). 


•  Satellites  do  not  fail  in  order  of  launch 

-  UHF  Follow-On  failures  on  orbit 

•  Flight  3  and  Flight  7 

-  UHF  Follow-On  operational  spacecraft 

•  Flight  2  and  Flight  4  (among  others) 
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CORPORATION 


•  Advantages:  Cost  (with  assumptions). 

•  Disadvantages: 

-  Fuel  budget 

-  TT&C  components  extended  life  requirement 

-  Additional  radiation 

-  Thermal/power  degredation 

-  Additional  ground  station  resources  required 


I  THE  AEROSPACE 


CORPORATION 


Cost  savings  wins  (with  assumptions). 
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Introduction 


♦  A  large  weapons  systems  project  had  a  need  for  a  Chief 
Software  Engineer  at  the  program  office  to  oversee  and 
manage  the  software  development  effort  of  several 
contractors. 

♦  The  project  was  incrementally  being  developed  with 
current  increment  in  the  design  phase  while  a  request  for 
proposal  was  being  developed  for  the  next  increment. 

♦  The  applications  have  critical  real-time  embedded 
command,  control,  and  communications  software  with 
many  interfaces  to  other  DoD  systems. 

♦  The  agency  asked  this  author  to  construct  a  list  of  the 
required  experience  and  skills  that  this  Chief  Engineer 
should  have  and  to  support  the  selection. 
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Introduction  (concluded) 


♦  This  position  is  critical  to  the  success  of  the  weapons  systems’ 
mission. 

♦  Software  is  key  in  this  success;  if  software  does  not  work,  the 
mission  fails. 

♦  Software  is  an  area  that  traditionally  has  not  received  the 
attention  that  it  deserves. 

♦  In  order  for  software  to  meet  mission  requirements  it  needs  to 
be  of  high  quality  and  maintainable,  developed  within  cost  and 
schedule,  and  managed  at  the  highest  professional  and 
technical  levels. 

♦  The  Project  Office  Software  Chief  Engineer  responsible  for  this 
has  to  have  the  appropriate  education,  experience  and  skills  at 
the  highest  possible  levels. 

♦  The  contents  of  this  presentation  can  be  used: 

>  In  other  organizations  looking  to  hire  a  Chief  Software  Engineer. 

>  To  increase  skills  of  Software  Engineers  in  the  Project  Office 
through  training. 
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Where  are  We? 


♦  Introduction 
i=^>  ♦  Qualification  Areas 


Education 

Configuration  Management 
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Risk  Management 

Project  Management 
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Proposals 
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Planning 
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Quality  Assurance 
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♦  Interviewing 

♦  Candidate  Evaluation 

♦  Summary 
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Qualification  Areas 


♦  This  position  requires  expertise  in  multiple  areas  of  software 
development,  including  technical,  acquisition,  and  management 
throughout  the  entire  life  cycle. 

♦  It  is  recognized  that  it  would  be  difficult  to  find  the  ideal 
candidate. 

>  A  selection  methodology  is  included  to  guide  the  selection  of  the 
best  possible  candidate. 

>  Gaps  in  the  qualification  areas  can  be  augmented  with  other 
individuals  in  the  program  office. 

♦  The  following  foils  present  these  qualification  areas  and 
describes  their  appropriate  attributes  for  this  position. 

♦  In  all  cases,  the  experience  is  relative  to  software-intensive 
systems,  preferably  embedded  and  real-time  weapon  systems. 
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Education 

(Qualification  Area) 

♦  A  degree  in  a  technical  discipline  (engineering,  computer  science) 
is  critical.  An  advanced  degree  (MS  or  Ph.D)  is  advantageous. 

♦  Additional  training  in  related  fields  is  a  benefit  (such  as  acquisition, 
networks,  radar,  etc.). 

♦  Training  in  specific  domain-applicable  technologies  is  also  a 
benefit. 

♦  Education  should  be  viewed  with  and  tempered  with  the 
experience  related  to  the  listed  qualification  areas. 
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Years  Experience 

(Qualification  Area) 

♦  Experience  in  large  software  intensive  development  efforts 
especially  for: 

>  Real-time,  embedded,  critical  weapons  systems  with  many 
interfacing  subsystems  with  multiple  contractors. 

♦  Experience  in  the  listed  qualification  areas  is  also  viewed  as 
important. 
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Management 

(Qualification  Area) 


♦  Experience  in  project  management  for  a  software  intensive 
system,  preferably  across  the  full  life  cycle. 

♦  Project  management,  program  management,  software 
management,  and  supervision  should  be  considered. 
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Proposal  Development  /  Evaluation 

(Qualification  Area) 


♦  Experience  in  developing  proposals  from  the  contractor  side. 

♦  Experience  in  writing  Requests  for  Proposal  (RFP)  and 
Statements-of  Work  (SOW). 

♦  Experience  in  evaluating  proposals  and  performing  source 
selection. 
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Planning 

(Qualification  Area) 

♦  Experience  in  planning  life  cycle  activities,  budgets,  schedules,  and 
resources  for  software  intensive  development  efforts  from  both  a 
development  and  acquisition  point-of-view. 

♦  Planning  should  include  developing  and  evaluating  plans  for 
conducting  the  activities  related  to  the  listed  qualification  areas. 
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Requirements 

(Qualification  Area) 

♦  Knowledge  of  the  nature  and  role  of  requirements  in  software 
intensive  systems. 

♦  Experience  in  gathering  user  needs,  translating  them  into 
technical  and  programmatic  requirements,  specifying,  verifying, 
validating  and  allocating  them  to  lower  levels  of  abstraction. 

♦  Experience  in  management  of  the  requirements  throughout  their 
entire  life  cycle. 
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Design 

(Qualification  Area) 

♦  Knowledge  of  software  design  techniques  and  tools. 

♦  Experience  in  the  design  of  software  intensive  systems  from 

>  conceptual  design, 

>  to  high  level  architecture, 

>  to  preliminary  design, 

>  to  detailed  design. 

♦  Ability  to  review  contractor  proposed  and  developed  design 
architectures. 
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Implementation 

(Qualification  Area) 

♦  Knowledge  of  key  programming  languages  (applicable  to  the 
domain  in  question). 

♦  Experience  in: 

>  coding  software  solutions, 

>  debugging, 

>  integrating  software  modules. 

♦  Ability  to  review  contractors’  implemented  code. 
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Test 

(Qualification  Area) 

♦  Knowledge  of  software  testing  techniques  and  tools. 

♦  Experience  in  the  formal  and  informal  testing  of  software 
intensive  systems,  ranging  from: 

>  unit  testing, 

>  integration  testing, 

>  formal  qualification  testing  (FQT), 

>  system  integration  tests, 

>  system  acceptance  tests, 

>  certification  tests. 

♦  Experience  in  the  development  of  test  plans,  test  descriptions, 
and  test  reports,  and  the  execution  of  the  tests. 
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Quality  Assurance 

(Qualification  Area) 

♦  Knowledge  of  software  quality  assurance  activities,  tools,  and 
techniques. 

♦  Experience  in  establishing  and  conducting  quality  assurance 
activities  for  large  software  programs,  with  a  focus  on  ensuring 
that  the: 

>  processes, 

>  procedures, 

>  standards 

that  are  used  on  the  project  are  followed  as  defined. 


16 


MITRE 


Configuration  Management 

(Qualification  Area) 

♦  Experience  in  establishing  and  conducting  configuration 
management  activities  for  large  software  programs. 

♦  Experience  in  baselining  requirements  and  managing  changes 
to  them. 

♦  Experience  in  conducting  impact  assessments  of  proposes 
changes 

♦  Experience  in  serving  on  configuration  control  boards. 
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Risk  Management 

(Qualification  Area) 


♦  Knowledge  of  risk  management  concepts. 

♦  Experience  in  establishing  and  conducting  risk  management 
activities  for  software  intensive  programs,  including: 

>  the  identification  of  project  risks, 

>  prioritizing  them, 

>  development  and  execution  of  mitigation  plans  and  alternatives 
(contingencies). 
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Metrics 

(Qualification  Area) 

♦  Knowledge  of  metrics  definition  and  application. 

♦  Experience  in  the: 

>  establishing  metrics  goals, 

>  collection  of  measurements  on  activities  and  products, 

>  definition  and  creation  of  metrics, 

>  analysis  of  resulting  metrics, 

>  actions  taken  based  on  the  analysis, 

>  reporting  of  resulting  findings. 
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Life  Cycle  Paradigms 

(Qualification  Area) 

♦  Knowledge  of  life  cycle  models  for  software  development, 
including  incremental,  evolutionary,  and  spiral. 

♦  Experience  in  defining  and  managing  a  software  intensive 
system  all  the  way  through  its  life  cycle,  from  operational 
concept  through  specification,  deployment  and  retirement. 

♦  Experience  in  defining  and  executing  entry  and  exit  criterion 
for  life  cycle  reviews  and  milestones 
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System  Engineering 

(Qualification  Area) 

♦  Knowledge  of  systems  engineering  practices  and  processes  for 
software  intensive  systems. 

♦  Experience  in  defining  and  applying  a  software  engineering 
process  within  a  systems  engineering  process. 
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Acquisition 

(Qualification  Area) 

♦  Experience  in  the  acquisition  of  software  intensive  systems. 

♦  Application  of  the  listed  qualification  areas  from  an  acquisition 
perspective. 

♦  Sponsor  of  specific  acquisition  processes. 

♦  The  ability  to  influence  others  in  the  importance  and  proper 
application  of  these  qualification  areas,  both  at  the  contractor 
and  program  office  level,  are  of  extreme  importance. 
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Standards 

(Qualification  Area) 

♦  Knowledge  of  and  experience  in  the  selection  and  application  of 
commercial  and  DoD  standards  to  complex  software-intensive 
systems. 

♦  Knowledge  of  the  role  of  standards  in  the  specification,  design  and 
development  of  large  software-intensive  systems. 

♦  Knowledge  of  sponsor  specific  standards  for  architecture, 
development,  management. 
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Process  Improvement 

(Qualification  Area) 

♦  Knowledge  of  the  Software  Engineering  Institute’s  (SEI) 
Capability  Maturity  Model  Integration  CMMI®. 

♦  Experience  in  measurement  of  process  effectiveness. 

♦  Experience  in  improvement  of  process  and  procedures  that  are 
followed  during: 

>  acquisition, 

>  development, 

>  operation 

of  software  intensive  systems. 


®  CMM  is  a  registered  trademark  of  the  SEI 
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Writing  Skills 

(Qualification  Area) 


♦  The  ability  to  write  both  technical  and  programmatic: 

>  reports, 

>  briefings, 

>  documents, 

>  plans, 

>  white  papers,  etc. 

in  a  clear,  understandable  and  concise  fashion. 
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Communication  Skills 

(Qualification  Area) 

♦  The  ability  to  communicate  with  management  and  technical 
individuals  in  a  clear,  understandable  and  concise  fashion. 

♦  The  ability  to  act  as  a  negotiator  between  the  contractor  and 
acquisition  organization. 
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Where  are  We? 


♦  Introduction 

♦  Qualification  Areas 


Education 

Configuration  Management 

Years  Experience 

Risk  Management 

Project  Management 

Metrics 

Proposals 

Life  Cycle  Paradigm 

Planning 

Systems  Engineering 

Requirements 

Acquisition 

Design 

Standards 

Implementation 

Process  Improvement 

Test 

Writing  Skills 

Quality  Assurance 

Communication  Skills 

♦  Interviewing 

♦  Candidate  Evaluation 

♦  Summary 

♦  Contact  Information 
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Interviewing 


♦  Prior  to  starting  the  interview  the  nature  of  the  project  and  the 
position  should  be  explained  to  the  candidate. 

♦  The  organization  and  project  should  be  explained  in  a  fashion 
that  entices  the  candidate  to  want  to  accept  an  offer. 

♦  The  importance  of  the  position  to  the  success  of  the  mission 
should  be  emphasized. 

♦  For  each  area  the  following  questions  should  be  asked  as  a 
minimum: 

>  Would  you  please  describe  your  experience  related  to 
(qualification  area) 

❖  How  much  of  this  experience  is  on  a  contractor  development 
effort? 

❖  How  much  of  this  experience  is  on  an  acquisition  effort? 

♦  Answers  to  these  and  other  questions  may  influence  what 
additional  questions  need  to  be  asked  for  that  area  or  other 
areas. 
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Interviewing  (concluded) 

♦  If  the  candidate  does  not  provide  the  needed  information, 
additional  questions  can  be  asked  in  an  attempt  to  elicit  the 
information. 

♦  Interview  notes  should  include  personal  style;  is  the  candidate: 

>  arrogant  or  personable,  poised  or  rattled? 

>  these  are  subjective  impressions  that  can  still  be  important  to  the 
interpersonal  aspects  of  his/her  job. 

♦  Additionally,  one  may  ask  the  candidate  to  provide  samples  of 
work/papers  written. 

♦  All  candidates  should  be  ranked  against  each  other  in  relation  to 
each  qualification  area. 

♦  At  least  two  interviewers  should  interview  each  candidate  to 
arrive  at  objective  evaluations. 

♦  The  following  foils  present  a  methodology  (an  example)  to  guide 
in  the  selection  of  the  best  possible  candidate 
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Example  -  Candidate  Evaluation 


Candidate  1 

Candidate  2 

Candidate  3 

Candidate  4 

Qualification  Area 

Weight 

Rank 

Score 

Rank 

Score 

Rank 

Score 

Rank 

Score 

Education 

6 

5 

30 

6 

36 

2 

12 

8 

48 

Years  Experience 

7 

6 

42 

9 

63 

7 

49 

6 

42 

Project  Management 

8 

8 

64 

6 

48 

5 

40 

8 

64 

Proposals 

7 

4 

28 

6 

42 

5 

35 

7 

49 

Planning 

8 

3 

24 

6 

48 

7 

56 

4 

32 

Requirements 

9 

6 

54 

9 

81 

6 

54 

7 

63 

Design 

5 

6 

30 

8 

40 

5 

25 

9 

45 

Implementation 

4 

8 

32 

7 

28 

6 

24 

5 

20 

Test 

7 

6 

42 

8 

56 

5 

35 

4 

28 

Quality  Assurance 

6 

7 

42 

6 

36 

8 

48 

6 

36 

Configuration  Management 

6 

4 

24 

8 

48 

9 

54 

7 

42 

Risk  Management 

8 

5 

40 

6 

48 

4 

32 

7 

56 

Metrics 

6 

6 

36 

7 

42 

7 

42 

9 

54 

Life  Cycle 

7 

5 

35 

7 

49 

5 

35 

7 

49 

Systems  Engineering 

8 

6 

48 

3 

24 

7 

56 

5 

40 

Acquisition 

10 

7 

70 

5 

50 

9 

90 

5 

50 

Standards 

7 

4 

28 

6 

42 

9 

63 

7 

49 

Process  Improvement 

9 

6 

54 

5 

45 

7 

63 

8 

72 

Writing  Skills 

7 

5 

35 

7 

49 

8 

56 

9 

63 

Communication  Skills 

8 

6 

48 

6 

48 

7 

56 

7 

56 

Total  Score 

806 

923 

925 

958 
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Candidate  Evaluation  (continued) 


♦  The  weight  of  each  qualification  area  indicates  the  importance  of 
a  particular  qualification  area  in  relation  to  all  other  qualification 
areas  and  depends  on  the  needs  of  the  organization. 

>  Weights  need  to  be  agreed  on  by  at  least  two  individuals  to  be 
objective  (could  be  management  of  the  interviewers). 

♦  Each  individual  is  ranked  against  each  other  on  all  qualification 
areas. 

♦  The  rank  of  each  individual  is  determined  by  at  least  two 
interviewers  to  be  objective. 

♦  The  score  is  the  product  of  the  weight  and  the  rank. 

♦  The  total  score  is  the  sum  of  all  scores. 
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Candidate  Evaluation  (concluded) 


♦  The  best  candidate  should  not  automatically  receive  a  10, 
experience  and  skills  against  the  area  should  be  the  major 
consideration. 

♦  The  total  score  is  the  sum  of  all  area  scores  which  are  the 
product  of  area  weight  and  candidates  rank  for  that  area. 

♦  The  maximum  total  score  is  1430. 

♦  Any  candidate  receiving  less  than  50%,  715,  should  not  be 
considered. 

♦  If  no  candidates  receive  at  least  50%,  a  new  round  of  interviews 
should  be  conducted. 

♦  When  scores  are  close,  a  judgement  call  may  be  necessary. 

♦  Interview  notes  on  personal  style  and  samples  of  work  can  be 
used  to  eliminate  candidates  or  to  select  from  among  those  with 
close  high  scores. 
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Summary 

♦  A  large  complex  weapons  systems  acquisition  effort  should 
have  an  experienced  software  chief  engineer  to  support  the 
effort;  the  experience  should  span  the  spectrum  of: 

>  Program/Project  Management 

>  Software  Engineering 

>  System  Engineering 

>  Test  Engineering 

>  Quality  Assurance 

>  Configuration  Management 

>  Risk  Analysis 

>  Metric  Analysis 

>  Life  Cycle  Activities 

>  Process  Engineering 

♦  This  experience  should  cover  both  supplier  development  efforts 
and  acquirer  acquisition  efforts. 

♦  The  criterion  was  successfully  used  to  select  a  Chief  Software 
Engineer. 
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Contact  Information 


Alfred  (Al)  W.  Florence 
florence@mitre.org 
(703)  983-7476 
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STRENGTH  THHOIGH  INDUSTRY  &  TECHNOLOGY 


SE  Effectiveness 

Overview 


The  SE  Effectiveness  Survey 

Quantifies  the  relationship  between  the 
application  of  Systems  Engineering  best  practices 
and  the  performance  of  system  development  projects 


Systems  Engineering 
capabilities  deliver 
better  Project 


TODAY’S  OUTLINE 

1.  The  Challenge 

2.  The  Rigor 

3.  The  Results! 

4.  Conclusions  &  Caveats 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


The  Challenge 

Stakeholder  Analysis 


Those  interested  in  such  a  study  -  and  their  interests 


Customers 

•  DoD  #1  SE  Issue  -  “Inconsistent  SE  Practices  across  life  cycle” 

•  Validate  initiatives  to  revitalize  SE 

•  Increase  emphasis  of  SE  content  in  RFPs  and  Proposals 
Industry  (System  Developers  &  Integrators) 

•  Proposal  may  skimp  on  SE;  Budget  pressures  on  SE 
Associations  &  Academia 

•  Unable  to  fully  satisfy  their  members  and  students 
SE  professionals 

•  Lack  rigorous  justification  for  their  recommendations 
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STRENGTH  THHOIGH  JRDLSTRY  &  TECHNOLOGY 


The  Challenge 

Previous  Studies  -  Partial  Insights 


Gruhl,  National  Avionics  and  Space  Administration  (NASA),  1992 

Compared  upfront  expenditures  to  eventual  cost  growth 

Herbsleb,  Software  Engineering  Institute  (SEI),  1994 

Studied  ROI  on  process  improvement  in  software 

Honour,  International  Council  on  Systems  Engineering  (INCOSE), 
2002 

Surveyed  industry  to  compare  SE  Effort  to  cost  &  schedule 

Valerdi  &  Boehm,  Constructive  System  Engineering  Cost  Model 
(COSYSMO),  2004 

Developed  parametric  estimation  model  similar  to  COCOMO 

Boehm  &  Valerdi,  SE  ROI  (COCOMO),  2006 

Analyzed  SE  activities  from  COCOMO  II 

Others... 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


The  Challenge 

Previous  Studies  -  Summary 


STUDY 

APPLICABILITY 

Author  & 
Background 

Findings 

SE  Activities 

Definition  of 
Success 

Characteristics 
of  Project 

Gruhl  (1992) 

32  NASA  Pgms 

8-15%  Upfront 
Best 

First  two  of  five 
development  phases 

Cost  (Less  cost 
overrun) 

Large;  Complex;  all 
NASA 

Herbsleb  (1994) 

13  CMM 
Companies 

Process 
Improvement 
ROI  4.0  —  8.8 

CMM  Process 

Areas 

Cost  (Cost 
reduction  through 

SE  investment) 

Various;  federal 
contracting 

Honour  (2004) 
Survey  INCOSE 
SEs 

15-20%  of 
project  should 
be  SE 

Overall  SE  level  of 
effort  (Cost)  & 
related  SE  quality 

Cost  &  Schedule 

Various  sizes 
(measured  by  total 
project  cost) 

Boehm  &  Valerdi 
(2006) 

COCOMO  II 

SE  importance 
grows  with 
project  size 

COCOMO  II  RESL 
(Architecture  and 
Risk) 

Cost 

Various  sizes,  but 
software  systems 
only 

Boehm  &  Valerdi 
(2004) 

COSYSMO 

Estimate 
within  30% 
effort  50%  - 
70%  of  time 

33  activities  defined 
by  EIA  632 

Cost 

Mostly  successful 
projects  from 
federal  contractors 

Ancona  & 

Caldwell  (1990) 

Boundary 

Management 

Managing  team 
boundary  15%; 
more  is  better 

Team  boundary 
activities  —  interface 
between  team  and 
external 

Product 

Performance 
(Successfully 
marketed  products) 

Technology 

products 

Frantz  (1995) 
Boeing  side-by- 
side  projects 

More  SE 
yielded  better 
quality  & 
shorter 
duration 

Defined  by  Frantz 

Product 

Performance  & 
Schedule  (Quality 
of  product  and 
duration  of  project) 

Three  similar 
systems  for 
manipulating 
airframes  during 
assembly 

Mink,  2007 
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NATIONAL  DITENSE  INDUSTRIAL  ASS 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


The  Rigor 

Followed^  Planned^  Lifecycle 


Form  Team 


Develop  Approach 
Validate  Approach 


NDIA 


Survey  (Industry  Projects) 


Pilot  Study  of  Survey 


Collect  Data 


Analyze  Data 


Anonymous  via  Web 
46+  Projects,  by  SEI 


Publish  Results 


Two  Reports: 

1.  Public  Report 

2.  Restricted 
Attachment 


This  study  spanned  three  years 
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STRENGTH  THHOIGH  INDUSTRY  &  TECHNOLOGY 


The  Rigor 

Formally  Selected  Set  of  SE  Activities 


CMMI-SW/SE  vi.i 

•  25  Process  Areas 

•  179  Goals 

•  614  Practices 

•  476  Work  Products 


Systems 
Engineering- 
related  Filter 


Size  Constraint 
Filter 


Considered  significant 
to  Systems  Engineering 


•  14  Process  Areas 

•  31  Goals 

•  87  Practices 

•  199  Work  Products 


•  1 3  Process  Areas 

•  23  Goals 

•  45  Practices 

•71  Work  Products 


Survey  was  developed  based  on  standards 
and  recognized  SE  experts 
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STRENGTH  THHOUGH  INDUSTRY  &  TECHNOLOGY 


The  Rigor 

Validated^  Survey  Responses 


Maximum  =  2.8 
3rd  Quartile  =  2.1 
Median  =  1.9 
1st  Quartile  =  1.7 
Minimum  =  1.1 
N  =  64 


Maximum  =  3.9 
3rd  Quartile  =  3.3 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  2.1 
N  =  63 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1.5 
N  =  64 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.75 
1st  Quartile  =  2.3 
Minimum  =  1.7 
N  =46 


Analyzed  distributions,  variability,  relationships... 
To  ensure  statistical  rigor  and  relevance 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Results 

Overall  SE  Capability  &  Project  Performance 


Overall  SE  Capability 


1 .00 


0.75 


0.50 


0.25 


0.00 


Lower  Moderate  Higher 


□  Higher  Project 
Performance 

□  Moderate  Project 
Peformance 

■  Lower  Project 
Performance 


Projects  with  better  Overall  Systems  Engineering 
Capability  delivers  better  Project  Performance 
(cost,  schedule  and  scope) 
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NATIONAL  DITENSE  INDUSTRIAL  ASS 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

1.  Product  Architecture  and  Project  Performance 


Product  Architecture  Capability 


NATIONAL  DITENSE  INDUSTRIAL  ASS 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

2.  Trade  Studies  and  Project  Performance 


Trade  Studies  Capability 


0 

o 

03 

E 

£ 

0 

□_ 


i.oo- 


0.75- 


0.50- 


0.25- 


0.00- 


17% 

17% 

44% 

41% 

39% 

42% 

49' 

H. 

32% 

19' 

H. 

Lower  Moderate  Higher 

Projects  with  better  Trade  Studies  Capability 
Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

3.  Technical  Solution  and  Project  Performance 


1.00- 


0.75- 

0.50- 

0.25- 

0.00- 


Technical  Solution  Capability 

4 


Lower  Moderate  Higher 


Projects  with  better  Technical  Solution  Capability 
Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 


NATIONAL  DITENSE  INDUSTRIAL  ASSOCIATION 


STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Results 

4.  IPTs  and  Project  Performance 


IPT  Capability 


1.00 


0 

o 

03 

E 

£ 

0 

□_ 


13% 


0.75- 


0.50- 


54% 


0.25  - 


0.00  -■ 


33% 


38% 

i 

431 

27% 


20% 


Lower  Moderate  Higher 


Projects  with  better  IPTs  Capability 
Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 


Performance 


NATIONAL  DITENSE  INDUSTRIAL  ASS 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

5.  Requirements  and  Project  Performance 


A  100-. 

0.75 

0.50 
0.25  J 
0.00 


Requirements  Capability 


18% 

21% 

55% 

38% 

53% 

44% 

18% 

26% 

27% 

Low 


Moderate 


High 


Projects  with  better  Requirements  Management  and  Development 
Capability  Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 


STRENGTH  THHOIGH  INDUSTRY  &  TECHNOLOGY 


Results 

Summary.  OL  Relationships 


SE  Capability' 

Architecture 
Trade  Studies 
Tech  Solution 
IPT 

Requirements 
Risk 
Validation 
Verification 
Prod  Integration 
Proj  Planning 
Config  Mngt 
Monitor  &  Ctrl 


Negative 

40%  -30%  -20%  -10% 


0%  10% 


Positive 

20%  30% 


40%  50% 


Relationship  to  Performance  (Gamma) 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Conclusions  &  Caveats 

Summary 


SE  Effectiveness 

•  Provides  credible  measured  evidence  about  the  value  of 
disciplined  Systems  Engineering 

•  Affects  success  of  systems-development  projects 

Specific  Systems  Engineering  Best  Practices 

•  Highest  relationships  to  activities  on  the  “left  side  of  SE  Vee” 

Environment  (Project  Challenge)  affects  performance  too 

-  Some  projects  are  more  challenging  than  others  ...  and  higher 
challenge  affects  performance  negatively 

-  Yet  good  SE  practices  remain  crucial  for  both  high  and  low 
challenge  projects 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Conclusions  &  Caveats 

Next  Steps 


•  Correlate  Report  Findings  with  Other  Sources 


•  Develop  Improvement  Recommendations 

•  Policy,  guidance,  training,  measures,  reviews 

•  Conduct  Additional  Analysis  of  Collected  Data 

•  iv  &  v 

•  Discover  other  relationships  and  correlations 

*  Repeat  the  Survey  to  Gauge  Improvements 


Survey  Acquirers 
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SE  Effectiveness 

Points  of  Contact 


Al  Brown 
Geoff  Draper 
Joe  Elm 

Dennis  Goldenson 
Al  Mink 
Ken  Ptack 


alan.r.brown2@boeing.com 

gdraper@harris.com 

jelm@sei.cmu.edu 

dg@sei.cmu.edu 

AI_Mink@SRA.com 

ken.ptack@ngc.com 


19 


SE  Effectiveness 
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Questions? 


Al  Mink 
al_mink@sra.  com 
571  212-4778 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Backup 


NDIA  SE  Effectiveness  Survey 
Analysis  Slides 


Reference:  “A  Survey  of  Systems  Engineering  Effectiveness”,  Software  Engineering  Institute,  Carnegie  Mellon  University, 
CMU/SEI-2007-SR-008.  Joseph  P.  Elm,  Dennis  R.  Goldenson,  Khaled  El  Emam,  Nicole  Donatelli,  Angelica  Nissa. 


Conclusions  &  Caveats 


NATIONAL  DITENSE  INDUSTRIAL  ASSOCIATION 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY  Consistent  with  ‘Top  10  Reasons  Projects  Fail*” 

1.  Lack  of  user  involvement 

2.  Changing  requirements 

3.  Inadequate  Specifications 

4.  Unrealistic  project  estimates 

5.  Poor  project  management 

6.  Management  change  control 

7.  Inexperienced  personnel 

8.  Expectations  not  properly  set 

9.  Subcontractor  failure 

10. Poor  architectural  design 


— 

Above  Items  Can  Cause  Overall 
Program  Cost  and  Schedule  to  Overrun 


*  Project  Management  Institute 


Matching  items  noted  in  RED  22 


Conclusions  &  Caveats 


NATIONAL  DITENSE  INDUSTRIAL  ASSOCIATION 


SHBEJSg-  Consistent  wjth  “Top  5  SE  Issues *”  (2006) 

•  Key  systems  engineering  practices  known  to  be  effective  are  not 
consistently  applied  across  all  phases  of  the  program  life  cycle. 

•  Insufficient  systems  engineering  is  applied  early  in  the  program  life 
cycle,  compromising  the  foundation  for  initial  requirements  and 
architecture  development. 

•  Requirements  are  not  always  well-managed,  including  the  effective 
translation  from  capabilities  statements  into  executable 
requirements  to  achieve  successful  acquisition  programs. 

•  The  quantity  and  quality  of  systems  engineering  expertise  is 
insufficient  to  meet  the  demands  of  the  government  and  the  defense 
industry. 

•  Collaborative  environments,  including  SE  tools,  are  inadequate  to 
effectively  execute  SE  at  the  joint  capability,  system  of  systems,  and 
system  levels. 


*  OUSD  AT&L  Summit 


Matching  items  noted  in  RED 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Summary  SE  Relationships 
to  Project  Performance 


Gamma 


Details 

Project  Challenge 

►J  PC  I  -31  %|  5~0%] 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

|  1.0  |  22%  |  28%  |  50%  |  1.85  | 


Relative  Project  Performance 


Moderate 

Min. 

Range 

Lo 

#  Med 

^ r 

Hi 

Max. 

Range 

|  42%  | 

|  58%  | 

Higher 

Min. 

Range 

Lo 

#  Med 

^ r 

Hi 

Max. 

Range 

r^~i 

00 

00 

|  38%  | 

|  25%  | 

Project  Environment 


CMMI 

22% 

13.0% 

IMP 

5% 

39.0% 

EXP 

9% 

33.0% 

1.0 

36% 

57% 

7% 

1.95 

1.0 

25% 

55% 

20% 

2.17 

1.0 

29% 

42% 

29% 

2.5 

2.7 

33% 

28% 

39% 

4.0 

2.84 

33% 

25% 

42% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

1.95 

29% 

36% 

35% 

2.7 

2.17 

42% 

29% 

29% 

2.84 

2.5 

39% 

44% 

17% 

3.5 

►I 

►I 

►I 

►j 

►I 

id 

id 

id 

id 

id 

id 

id 

id 

id 


Systems  Engineering  Capability 
IPT 
PP 
PMC 
RSKM 
REQ 
TRADE 
ARCH 
TS 
PI 

VER 
VAL 
CM 

Overall  SEC 
REQ+TS 


1.0 

33% 

54% 

13% 

2.5 

1.0 

33% 

54% 

13% 

2.8 

1.0 

23% 

54% 

23% 

2.5 

1.0 

35% 

47% 

18% 

2.8 

1.0 

44% 

38% 

18% 

2.8 

1.0 

39% 

44% 

17% 

2.7 

1.0 

45% 

44% 

11% 

2.7 

1.0 

40% 

53% 

7% 

2.8 

1.0 

36% 

54% 

14% 

1.5 

1.0 

31% 

62% 

7% 

2.7 

1.0 

54% 

23% 

23% 

2.7 

1.0 

29% 

47% 

24% 

3.0 

1.0 

39% 

46% 

15% 

2.5 

1.0 

43% 

50% 

13% 

2.8 

2.5 

43% 

38% 

19% 

3.1 

2.8 

29% 

35% 

36% 

3.3 

2.5 

23% 

46% 

31% 

3.0 

2.8 

27% 

66% 

7% 

3.6 

2.8 

26% 

53% 

21% 

3.4 

2.7 

42% 

41% 

17% 

3.3 

2.7 

29% 

42% 

29% 

3.3 

2.8 

33% 

40% 

27% 

3.2 

1.5 

33% 

38% 

29% 

3.5 

2.7 

33% 

34% 

33% 

3.2 

2.7 

17% 

66% 

17% 

3.3 

3.0 

46% 

36% 

18% 

3.67 

2.5 

29% 

59% 

12% 

3.0 

2.8 

23% 

62% 

15% 

3.1 

3.1 

20% 

27% 

53% 

4.0 

3.3 

35% 

29% 

36% 

4.0 

3.0 

45% 

25% 

30% 

4.0 

3.6 

36% 

0% 

64% 

4.0 

3.4 

27% 

18% 

55% 

4.0 

3.3 

19% 

32% 

49% 

4.0 

3.3 

23% 

31% 

46% 

4.0 

3.2 

27% 

27% 

46% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

3.2 

33% 

20% 

47% 

4.0 

3.3 

29% 

33% 

38% 

4.0 

3.67 

28% 

33% 

39% 

4.0 

3.0 

31% 

13% 

56% 

4.0 

3.1 

22% 

28% 

50% 

4.0 

Acquirer  Capability 

AC  |  -35% |  3~0%1  |  1.0  |  7%|  60% |  33%|  2.5  | 


|  2.5  |  41  %|  32% |  26% |  3.0  | 


|  3.0  |  50% |  25% |  25% |  4.0  | 


Combined  Capability  and  Challenge 

►  |  REQ+TS+PC  |  63%|  0~0%]  |  1.0  |  67%|  33%|  0%|  1.7  | 


|  1.7  |  25% |  45% |  30%|  2.3  | 


|  2.3  |  14%  |  36% |  50% |  4.0  | 


Gamma  relationship  Chance  probability  _ Gamma  relationship  Chance  probability 


Strong 

Very  low 

|  |  Moderately  strong 

Moderately  low 

Moderately  strong 

Low 

|  |  Weak 

Fair 

to  strong 
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Summary  SE  Relationships 
to  Project  Performance 


Relative  Project  Performance 


Gamma 


Details 

Project  Challenge 

►i  pc  r 


Project  Environment 


►I 

►I 

►I 

►I 

id 

id 

id 

id 

id 

■ 

id 

id 

id 

id 


CMMI 

22% 

13.0% 

IMP 

5% 

39.0% 

EXP 

9% 

33.0% 

Systems  Engineering  Capability 

IPT 

34% 

4.0% 

PP 

13% 

25.0% 

PMC 

-13% 

25.0% 

RSKM 

28% 

6.1% 

REQ 

33% 

4.0% 

TRADE 

37% 

3.0% 

ARCH 

40% 

0.2% 

TS 

36% 

3.0% 

PI 

21% 

16.0% 

VER 

25% 

9.0% 

VAL 

28% 

7.0% 

CM 

13% 

26.0% 

Overall  SEC 

32% 

4.0% 

REQ+TS 

49% 

0.5% 

Acquirer  Capability 

C 


AC 


-35 


Combined  Capability  am 
►|  REQ+TS+PC  P 


63 


Lower 

Min. 

^ r 

W 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

Moderate 

Min. 

Range 

Lo 

#  Med 

Hi 

Max. 

Range 

hid 

|  42%  | 

|  58%  | 

Higher 

Min. 

Range 

Lo 

#  Med 

^ r 

Hi 

Max. 

Range 

r^~i 

00 

00 

o 

00 

CO 

|  25%  | 

1.0 

36% 

1.0 

25% 

1.0 

29% 

Highest  scoring  SE  capability  areas  in  Higher  Performing  Projects*: 
Risk  Management;  Requirements  Development  and  Management;  IPTs 

|  *Based  on  small  partitioned  sample  size 


1.0 

33% 

54% 

13% 

2.5 

1.0 

33% 

54% 

13% 

2.8 

1.0 

23% 

54% 

23% 

2.5 

1.0 

35% 

47% 

18% 

2.8 

1.0 

44% 

38% 

18% 

2.8 

1.0 

39% 

44% 

17% 

2.7 

1.0 

45% 

44% 

11% 

2.7 

1.0 

TW 

53% 

7% 

2.8 

1.0 

36% 

54% 

14% 

1.5 

1.0 

31% 

62% 

7% 

2.7 

1.0 

54% 

23% 

23% 

2.7 

1.0 

29% 

47% 

24% 

3.0 

1.0 

39% 

46% 

15% 

2.5 

1.0 

43% 

50% 

13% 

2.8 

2.5 

43% 

38% 

19% 

3.1 

2.8 

29% 

35% 

36% 

3.3 

2.5 

23% 

46% 

31% 

3.0 

2.8 

27% 

66% 

7% 

3.6 

2.8 

26% 

53% 

21% 

3.4 

2.7 

42% 

41% 

17% 

3.3 

2.7 

29% 

42% 

29% 

3.3 

2.8 

33% 

40% 

27% 

3.2 

1.5 

33% 

38% 

29% 

3.5 

2.7 

33% 

34% 

33% 

3.2 

2.7 

17% 

66% 

17% 

3.3 

3.0 

46% 

36% 

18% 

3.67 

2.5 

29% 

59% 

12% 

3.0 

2.8 

23% 

62% 

15% 

3.1 

Lowest  scoring  SE  capability  areas  in  Lower  Performing  Projects*: 
Validation;  Architecture;  Requirements  Development  and  Management 


3.1 

20% 

27% 

53% 

4.0 

3.3 

35% 

29% 

36% 

4.0 

3.0 

45% 

25% 

30% 

4.0 

3.6 

36% 

0% 

64% 

4.0 

3.4 

27% 

18% 

55% 

4.0 

3.3 

19% 

32% 

49% 

4.0 

3.3 

23% 

31% 

46% 

4.0 

3.2 

27% 

27% 

46% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

3.2 

33% 

20% 

47% 

4.0 

3.3 

29% 

33% 

38% 

4.0 

3.67 

28% 

33% 

39% 

4.0 

3.0 

31% 

13% 

56% 

4.0 

3.1 

22% 

28% 

50% 

4.0 

|  25% |  25% |  4.0  | 


|  36% |  50%|  4.0  | 


Gamma  relationship  Chance  probability  _ Gamma  relationship  Chance  probability 


Strong 

Very  low 

|  |  Moderately  strong 

Moderately  low 

Moderately  strong 

Low 

|  |  Weak 

Fair 

to  strong 
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STRENGTH  THHtKGH  INDUSTRY  &  TECHNOLOGY 


Terminology  and  Notation 

Distribution  Graph 


Histogram  of 
response 
frequencies 


Outliers 


Interquartile 
Range 


Median 


Maximum  =  3.8 
3rd  Quartile  =  3.2 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1.0 
N  =  64 


V. 


“"V- 

Data 

Range 


Sample  size 

(responses  to  corresponding 
survey  questions) 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Terminology  and  Notation 

Mosaic  Chart 


Column  width 
represents  proportion 
of  projects  with 
this  level  of 


(Lowest,  Intermediate,  Highest) 


Measures  of 


Sample  size  and  distribution  for 
associated  survey  responses 
(capability  +  performance) 


association 
and  statistical  test 


Relative  performance 
distribution  of  the 
sample 


Gamma:  measures  strength  of 
relationship  between  two  ordinal 
variables 

probability  that  an  associative 
relationship  would  be  observed 
by  chance  alone 


STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Product  Architecture  (ARCH) 


Maximum  =  4.0 
3rd  Quartile  =  3.5 
Median  =  2.8 
1st  Quartile  =  2.6 
Minimum  =  2.0 
N  =  57 


1.00 


0.75 


0.50 


0.25 


0.00 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(x*2.7)  (2.7<x<  3.3)  (x*3.3) 

N  =  18  N  = 1 4  N  = 1 3 


Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x  £  3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.40 
p  =  0.002 


Relationship  to  project  performance: 


Moderately  strong  to  strong  positive 
relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

45% 

44% 

11% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

23% 

31% 

46% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

29% 

42% 

29% 

3.3 

Gamma 

P 

40% 

0.2% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Product  Architecture  (ARCH) 


Survey  Questions 


ID 

Question 

Response  range 

IF01 

This  project  maintains  accurate  and  up-to-date  descriptions  (e.g.  interface  control 
documents,  models,  etc.)  defining  interfaces  in  detail 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF02 

Interface  definition  descriptions  are  maintained  in  a  designated  location,  under 
configuration  management,  and  accessible  to  all  who  need  them 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03a 

For  this  project,  the  product  high-level  structure  is  documented,  kept  up  to  date,  and 
managed  under  configuration  control 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03b 

For  this  project,  the  product  high-level  structure  is  documented  using  multiple  views  (e.g. 
functional  views,  module  views,  etc. 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03c 

For  this  project,  the  product  high-level  structure  is  accessible  to  all  relevant  project 
personnel 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF04 

This  project  has  defined  and  documented  guidelines  for  choosing  COTS  product 
components 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Configuration  Management  (CM) 


Maximum  =  4.0 
3rd  Quartile  =  4.0 
Median  =  3.6 
1st  Quartile  =  3.0 
Minimum  =  2.0 
N  =  58 


Lower 

Capability 

(«3) 

N=  17 


Moderate 
Capability 
(3  <  x  <  4) 
N  =  1 1 


Higher 
Capability 
(x=4) 
N=  18 


Gamma  =  0.1 3 

p  =  0.26 


Relationship  to  project  performance: 


Weak  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

29% 

47% 

24% 

3.0 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.67 

28% 

33% 

39% 

4.0 

30 


Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.0 

46% 

36% 

18% 

3.67 

Gamma 

P 

13% 

26.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

Configuration  Management  (CM) 


ID 

Question 

Response  Range 

V&V06 

This  project  has  a  configuration  management  system  that  charters  a  Change  Control 

Board  to  disposition  change  requests 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V07 

This  project  maintains  records  of  requested  and  implemented  changes  to  configuration- 
managed  items 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V08 

This  project  creates  and  manages  configuration  baselines  (e.g.,  functional,  allocated, 
product) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

IPT-Related  Capability  (IPT) 


Maximum  =  4.0 
3rd  Quartile  =  3.5 
Median  =  3.0 
1st  Quartile  =  2.5 
Minimum  =  1.0 
N  =  64 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(X£  2.5)  (2.7  <  X  <  3.1)  (xs3.1) 

N=15  N=16  N=15 


Best  Performance 
(x  >  3.0) 


Moderate 
Performance 
(2.5  <  x^3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.34 
p=  0.04 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

33% 

54% 

13% 

2.5 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.1 

20% 

27% 

53% 

4.0 

32 


Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.5 

43% 

38% 

19% 

3.1 

Gamma 

P 

34% 

4.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

IPT-Related  Capability  (IPT) 


ID 

Question 

Response  range 

Proj03 

This  project  uses  integrated  product  teams  (IPTs) 

•Yes 

•No 

Proj04 

This  project  makes  effective  use  of  integrated  product  teams  (IPTs) 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 

Proj06 

My  suppliers  actively  participate  in  IPTs 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 

ProjOYa 

This  project  has  an  IPT  with  assigned  responsibility  for  systems  engineering 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 

ProjOYb 

This  project  has  Systems  Engineering  representation  on  each  IPT 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Product  Integration  (PI) 


Maximum  =  4.0 
3rd  Quartile  =  3.0 
Median  =  3.0 
1st  Quartile  =  2.0 
Minimum  =  2.0 
N  =  57 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(x  =  1)  (x=  2  or  3)  (x=  4) 

N  =  1  4  N  =  24  N  =  7 


Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x^3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.21 
p=  0.16 


Relationship  to  project  performance: 


Weak  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

36% 

54% 

14% 

1.5 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.5 

29% 

29% 

42% 

4.0 

34 


Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.5 

33% 

38% 

29% 

3.5 

Gamma 

P 

21% 

16.0% 

STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Survey  Question 


SE  Capability: 

Product  Integration  (PI) 


ID 

Question 

Response  range 

IF05 

This  project  has  accurate  and  up-to-date  documents  defining  its  product  integration 
process,  plans,  criteria,  etc.  throughout  the  life  cycle 

•strongly  disagree  t 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Monitoring  and  Control  (PMC) 


Maximum  =  3.8 
3rd  Quartile  =  3.2 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1.0 
N  =  64 


Lower 
Capability 
(xs  2.5) 
N  =  13 


Moderate 
Capability 
(2.5  <x  <  3) 
N  =  1  3 


Higher 
Capability 
(x*3) 
N=  20 


Gamma  =  -0.1 3 
p=  0.25 


Relationship  to  project  performance:  Weak  negative  relationship 


SE  Capability 


PMC 


Gamma 

P 

-13% 

25.0% 

Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

23% 

54% 

23% 

2.5 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.5 

23% 

46% 

31% 

3.0 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.0 

45% 

25% 

30% 

4.0 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Monitoring  and  Control  (PMC) 


Survey  Questions  (Part  1) 


ID 

Question 

Response  range 

Conti  3 

Do  you  separately  cost  and  track  systems  engineering  activities? 

Yes 

No 

Conti  4a 

Approximately  what  percentage  of  non-recurring  engineering  (NRE)  does  systems 
engineering  represent? 

Percentages  quantized  as: 

•<=  5% 

•<=  10% 

•<=  15% 

•<=  25% 

•>  25% 

Conti  4b 

Is  the  NRE  percentage  estimated,  or  is  it  a  measured  value? 

•estimated 

•measured 

PerfOl 

This  project  creates  and  manages  cost  and  schedule  baselines 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

Perf02b 

EVMS  data  are  available  to  decision  makers  in  a  timely  manner  (i.e.  current  within  2 
weeks) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

Perf02c 

The  requirement  to  track  and  report  EVMS  data  is  levied  upon  the  project’s  suppliers 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

Perf02d 

Variance  thresholds  for  CPI  and  SPI  variance  are  defined,  documented,  and  used  to 
determine  when  corrective  action  is  needed 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Monitoring  and  Control  (PMC) 


Survey  Questions  (Part  2) 


ID 

Question 

Response  range 

Perf02e 

EVMS  is  linked  to  the  technical  effort  through  the  WBS  and  the  IMP/IMS 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

OPerf05 

Does  this  project  track  reports  of  problems  from  fielded  items? 

•Yes 

•No 

Scored  by  • 

the  number 
of  positive 
responses 

OPerf06 

Does  the  project  conduct  an  engineering  assessment  of  all  field  trouble  reports? 

•Yes 

•No 

OPerfOY 

The  results  of  this  engineering  assessment  feed  into  ... 

•operational  hazard 

risk  assessments 

•materiel  readiness 

assessments 

•system  upgrades 

planning 

•other 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Planning  (PP) 


i 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.6 
Minimum  =  2.0 
N  =  63 


1.00- 


0.75- 


0.50- 


0.25- 


0.00-' 


13% 

36% 

36% 

54% 

35% 

29% 

33% 

35% 

29% 

Lower  Moderate  Higher 

Capability  Capability  Capability 

(x^  2.8)  (2.8<  x  <  3.3)  (x ^  3.3) 

N  =  1 5  N  =  1 4  N  =  1 7 


Best 

Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x  <  3.0) 

Lower 

Performance 
(x  <  2.5) 


Gamma  =  0.13 
p  =  0.25 


Relationship  to  project  performance: 


Weak  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

33% 

54% 

13% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

35% 

29% 

36% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

29% 

35% 

36% 

3.3 

Gamma 

P 

13% 

25.0% 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  1) 


SE  Capability: 

Project  Planning  (PP) 


ID 

Question 

Response  range 

PD01 

This  project  utilizes  a  documented  set  of  systems  engineering  processes  for  the  planning 
and  execution  of  the  project 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02a 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
includes  task  descriptions  and  work  package  descriptions 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02b 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that  is 
based  upon  the  product  structure 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02c 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that  is 
developed  with  the  active  participation  of  those  who  perform  the  systems  engineering 
activities 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02d 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that  is 
developed  with  the  active  participation  of  all  relevant  stakeholders,  e.g.,  developers, 
maintainers,  testers,  inspectors,  etc. 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD03a 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and  methodology  to  create 
the  initial  conceptual  design  for  product  development)  is  complete,  accurate  and  up-to- 
date 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD03b 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and  methodology  to  create 
the  initial  conceptual  design  for  product  development)  is  developed  with  the  active 
participation  of  those  who  perform  the  systems  engineering  activities 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD03c 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and  methodology  to  create 
the  initial  conceptual  design  for  product  development)  is  developed  with  the  active 
participation  of  all  appropriate  functional  stakeholder 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Planning  (PP) 


Survey  Questions  (Part  2) 


ID 

Question 

Response  range 

PD04a 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is  an 
event-driven  plan  (i.e. ,  each  accomplishment  is  tied  to  a  key  project  event) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD04b 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that 
documents  significant  accomplishments  with  pass/fail  criteria  for  both  business  and 
technical  elements  of  the  project 

•strongly  disagree  j 

•disagree 

•agree 

•strongly  agree 

PD04c 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is 
consistent  with  the  WBS 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05a 

This  project  has  an  integrated  event-based  schedule  that  is  structured  as  a  networked, 
multi-layered  schedule  of  project  tasks  required  to  complete  the  work  effort 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05b 

This  project  has  an  integrated  event-based  schedule  that  contains  a  compilation  of  key 
technical  accomplishments  (e.g.,  a  Systems  Engineering  Master  Schedule) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05c 

This  project  has  an  integrated  event-based  schedule  that  references  measurable  criteria 
(usually  contained  in  the  Integrated  Master  Plan)  required  for  successful  completion  of 
key  technical  accomplishments 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05d 

This  project  has  an  integrated  event-based  schedule  that  is  consistent  with  the  WBS 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Planning  (PP) 


Survey  Questions  (Part  3) 


ID 

Question 

Response  range 

PD05e 

This  project  has  an  integrated  event-based  schedule  that  identifies  the  critical  path  of  the 
program  schedule 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD06 

This  project  has  a  plan  or  plans  for  the  performance  of  technical  reviews  with  defined 
entry  and  exit  criteria  throughout  the  life  cycle  of  the  project 

•strongly  disagree  j 

•disagree 

•agree 

•strongly  agree 

PD07 

This  project  has  a  plan  or  plans  that  include  details  of  the  management  of  the  integrated 
technical  effort  across  the  project  (e.g.,  a  Systems  Engineering  Management  Plan  or  a 
Systems  Engineering  Plan) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD08 

Those  who  perform  systems  engineering  activities  actively  participate  in  the  development 
and  updates  of  the  project  planning 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD09 

Those  who  perform  systems  engineering  activities  actively  participate  in 
tracking/reporting  of  task  progress 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHOLGH  INDUSTRY  &  TCCHNOLOGY 


SE  Capability: 

Requirements  Development  &  Mgmt  (REQ) 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.8 
Minimum  =  2.2 
N  =  58 


1.00_ 

0.75 

0.50_ 
0. 25  _ 
0.00 


18% 

21% 

55% 

38% 

53% 

44% 

18% 

26% 

27% 

Lower 

Moderate 

High 

Capability 

Capability 

Capat 

(x  <  2.S) 

(2.0  <x  <  3.4) 

(x  >: 

N=  16 

N=  19 

N  =  ■ 

Best  Performance 
(x  >3.0) 

Moderate 
Performance 
(25  <  x  <3.01 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.33 
p=  0.04 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

44% 

38% 

18% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.4 

27% 

18% 

55% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

26% 

53% 

21% 

3.4 

Gamma 

P 

33% 

4.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Requirements  Development  &  Mgmt  (REQ) 


Survey  Questions  (Part  1) 


ID 

Question 

Response  range 

RDOIa 

This  project  maintains  an  up-to-date  and  accurate  listing  of  all  requirements  specified  by 
the  customer,  to  include  regulatory,  statutory,  and  certification  requirements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RDOlb 

This  project  maintains  an  up-to-date  and  accurate  listing  of  all  requirements  derived  from 
those  specified  by  the  customer 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD02 

This  project  maintains  up-to-date  and  accurate  documentation  clearly  reflecting  the 
hierarchical  allocation  of  both  customer  and  derived  requirements  to  each  element 
(subsystem,  component,  etc.)  of  the  system  in  the  configuration  baselines 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD03a 

This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of 
operational  concepts  and  their  associated  scenarios 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD03b 

This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  use  cases 
(or  their  equivalent) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD03c 

This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  product 
installation,  maintenance  and  support  concepts 

•strongly  disagree  \ 

•disagree 

•agree 

•strongly  agree 

RD04 

This  project  has  documented  criteria  for  identifying  authorized  requirements  providers  to 
avoid  requirements  creep  and  volatility 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Requirements  Development  &  Mgmt  (REQ) 


Survey  Questions  (Part  2) 


ID 

Question 

Response  range 

RD05 

This  project  has  documented  criteria  (e.g.,  cost  impact,  schedule  impact,  authorization  of 
source,  contract  scope,  requirement  quality)  for  evaluation  and  acceptance  of 
requirements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD06 

The  requirements  for  this  project  are  approved  in  a  formal  and  documented  manner  by 
relevant  stakeholders 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD07 

This  project  performs  and  documents  requirements  impact  assessments  for  proposed 
requirements  changes 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD08 

This  project  develops  and  documents  project  requirements  based  upon  stakeholder 
needs,  expectations,  and  constraints 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD09 

This  project  has  an  accurate  and  up-to-date  requirements  tracking  system 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RDlOa 

For  this  project,  the  requirements  documents  are  managed  under  a  configuration  control 
process 

•strongly  disagree  \ 

•disagree 

•agree 

•strongly  agree 

RDlOb 

For  this  project,  the  requirements  documents  are  accessible  to  all  relevant  project  staff 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Risk  Management  (RSKM) 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.8 
Minimum  =  2.2 
N  =  58 


Lower 
Capability 
(xs  2.8) 
N=  17 


0% 

1/ 


Moderate  Higher 

Capability  Capability 

(2.7  <  K  <  3.6)  (xs3.6) 

N=15  N  =  1  4 


Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x£3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.28 

p  =  0.061 


Relationship  to  project  performance:  Moderately  strong  positive  relationship 


SE  Capability 


RSKM 


Gamma 

P 

28% 

6.1% 

Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

35% 

47% 

18% 

2.8 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

27% 

66% 

7% 

3.6 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.6 

36% 

0% 

64% 

4.0 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

Risk  Management  (RSKM) 


ID 

Question 

Response  range 

PDIIa 

This  project  has  a  Risk  Management  process  that  creates  and  maintains  an  accurate  and 
up-to-date  list  of  risks  affecting  the  project  (e.g.,  risks  to  cost,  risks  to  schedule,  risks  to 
performance) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PDIIb 

This  project  has  a  Risk  Management  process  that  creates  and  maintains  up-to-date 
documentation  of  risk  mitigation  plans  and  contingency  plans  for  selected  risks 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PDIIc 

This  project  has  a  Risk  Management  process  that  monitors  and  reports  the  status  of  risk 
mitigation  activities  and  resources 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PDIId 

This  project  has  a  Risk  Management  process  that  assesses  risk  against  achievement  of 
an  event-based  schedule 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD12 

This  project's  Risk  Management  process  is  integrated  with  program  decision-making 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Trade  Studies  (TRADE) 


Maximum  =  4.0 
3rd  Quartile  =  3.7 
Median  =  3.0 
1st  Quartile  =  2.3 
Minimum  =  1.0 
N  =  58 


1.00 


0.75 


0.50 


0.25 


0.00 


Best  Performance 
(x  >  3.0) 


Moderate 
Performance 
(2.5  <  x£  3.0) 

Lower  Performance 
(x  <  2.5) 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(x^  2.7)  (2.7  <  x  <  3.3)  (xa3.3) 

N  =  1  8  N  =  1  2  N=16 


Gamma  =  0.37 
p  =  0.03 


Relationship  to  project  performance: 


Moderately  strong  to  strong  positive 
relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

39% 

44% 

17% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

19% 

32% 

49% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

42% 

41% 

17% 

3.3 

Gamma 

P 

37% 

3.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

Trade  Studies  (TRADE) 


ID 

Question 

Response  range 

RD11 

Stakeholders  impacted  by  trade  studies  are  involved  in  the  development  and 
performance  of  those  trade  studies 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD12 

This  project  performs  and  documents  trade  studies  between  alternate  solutions  based 
upon  definitive  and  documented  selection  criteria 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD13 

Documentation  of  trade  studies  is  maintained  in  a  defined  repository  and  is  accessible  to 
all  relevant  project  staff 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHtKGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Technical  Solution  (TS) 


Maximum  =  4.0 
3rd  Quartile  =  3.3 
Median  =  2.9 
1st  Quartile  =  2.6 
Minimum  =  2.1 
N  =  57 


Note:  TS  is  a  composite  measure  equivalent  to  ARCH  +  TRADE. 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(xs2.8)  (2.8  <  x  <  3.2)  (xa3.2) 

N=15  N  =  1  5  N  =  1  5 


Gamma  =  0.36 
p  =  0.03 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

40% 

53% 

7% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.2 

27% 

27% 

46% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

33% 

40% 

27% 

3.2 

Gamma 

P 

36% 

3.0% 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  1) 


SE  Capability: 

Technical  Solution  (TS) 


ID 

Question 

Response  Range 

RD11 

Stakeholders  impacted  by  trade  studies  are  involved  in  the  development  and 
performance  of  those  trade  studies 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD12 

This  project  performs  and  documents  trade  studies  between  alternate  solutions  based 
upon  definitive  and  documented  selection  criteria 

•strongly  disagree  ' 

•disagree 

•agree 

•strongly  agree 

RD13 

Documentation  of  trade  studies  is  maintained  in  a  defined  repository  and  is  accessible  to 
all  relevant  project  staff 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF01 

This  project  maintains  accurate  and  up-to-date  descriptions  (e.g.  interface  control 
documents,  models,  etc.)  defining  interfaces  in  detail 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF02 

Interface  definition  descriptions  are  maintained  in  a  designated  location,  under 
configuration  management,  and  accessible  to  all  who  need  them 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  2) 


SE  Capability: 

Technical  Solution  (TS) 


ID 

Question 

Response  Range 

IF03a 

For  this  project,  the  product  high-level  structure  is  documented,  kept  up  to  date,  and 
managed  under  configuration  control 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03b 

For  this  project,  the  product  high-level  structure  is  documented  using  multiple  views  (e.g. 
functional  views,  module  views,  etc.) 

•strongly  disagree  ' 

•disagree 

•agree 

•strongly  agree 

IF03c 

For  this  project,  the  product  high-level  structure  is  accessible  to  all  relevant  project 
personnel 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF04 

This  project  has  defined  and  documented  guidelines  for  choosing  COTS  product 
components 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Validation  (VAL) 


Maximum  =  4.0 
3rd  Quartile  =  3.7 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  1.7 
N  =  58 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(xs2.7)  (2.7  <  x  <  3.3)  (x ^  3.3) 

N  =  1  3  N  =  1  2  N  =  21 


Gamma  =  0.28 
p  =  0.07 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

54% 

23% 

23% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

29% 

33% 

38% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

17% 

66% 

17% 

3.3 

Gamma 

P 

28% 

7.0% 

VAL 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Validation  (VAL) 


Survey  Questions 


ID 

Question 

Response  Rate 

V&V04a 

This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
validation  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V04b 

This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used 
for  the  validation  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V05 

This  project  maintains  a  listing  of  items  managed  under  configuration  control 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

54 


STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Verification  (VER) 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.6 
Minimum  =  2.2 
N  =  58 


7% 


Lower 
Capability 
0^2.7) 
N  =  16 


Moderate  Higher 

Capability  Capability 

(2.7  <  x  <  3.2)  (x^  3.2) 

N=15  N=15 


Gamma  =  0.25 
p=  0.09 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

31% 

62% 

7% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.2 

33% 

20% 

47% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

33% 

34% 

33% 

3.2 

Gamma 

P 

25% 

9.0% 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Verification  (VER) 


Survey  Questions  (Part  1) 


ID 

Question 

Response  range 

V&VOIa 

This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
test  and  verification  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&VOlb 

This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used  for 
the  verification  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02a 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  defines  entry  and  exit  criteria  for  work  products 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02b 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  includes  training  requirements  for  the  reviewers 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02e 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  addresses  identified  risks  and  risk  mitigation  activities  during  reviews 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02f 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  examines  completeness  of  configuration  baselines 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  2) 


SE  Capability: 

Verification  (VER) 


ID 

Question 

Response  range 

V&V03 

This  project  conducts  non-advocate  reviews  (e.g.  reviews  by  qualified  personnel  with  no 
connection  to  or  stake  in  the  project)  and  documents  results,  issues,  action  items,  risks, 
and  risk  mitigations 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02C 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  defines  criteria  for  the  selection  of  work  products  (e.g.,  requirements 
documents,  test  plans,  system  design  documents,  etc.)  for  review 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02d 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  tracks  action  items  to  closure 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  TUHOLGH  INDUSTRY  &  TCCHNOLOGY 


SE  Capability: 

Combined  Reqts+Tech  Solution  (REQ+TS) 


(This  is  a  higher  order  measure; 
see  base  measures  for  distribution) 


Lower  Moderate  Best 

Capability  Capability  Capability 

(xs2.8)  (2.8  <  x  <  3.1)  (x^  3.1) 

N  =  1  5  N  =  1  3  N  =  1  8 


Gamma  =  0.49 
p  =  0.005 


Relationship  to  project  performance: 


Strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

43% 

50% 

13% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.1 

22% 

28% 

50% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

23% 

62% 

15% 

3.1 

Gamma 

P 

49% 

0.5% 

REQ+TS 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Total  Systems  Engineering  Capability 


Maximum  =  3.9 
3rd  Quartile  =  3.3 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  2.1 
N  =  63 


1.00- 


0.75- 


0.50- 


0.25  - 


0.00- 


46% 


Lower 
Capability 
(x  £2.5) 
N  =  13 


12% 

ii - 

59% 

56% 

13% 

29% 

31% 

Moderate 
Capability 
(2.5<x<3.0) 
N  =  17 

Higher 
Capability 
(x  *3.0) 

N  =  16 

Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x£3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.32 
p  =  0.04 


Relationship  to  project  performance:  Moderately  strong  positive  relationship 


SE  Capability 


Overall  SEC 


Gamma 

P 

32% 

4.0% 

Lower 

Min. 

^ r 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

39% 

46% 

15% 

2.5 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.5 

29% 

59% 

12% 

3.0 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.0 

31% 

13% 

56% 

4.0 

59 


STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Project  Challenge  (PC) 


■ 

Project  challenge  factors: 

•Life  cycle  phases 

•Project  characteristics 

(e.g.,  size,  effort,  duration,  volatility) 

•Technical  complexity 

•Teaming  relationships 


Lower  Moderate  Higher 

Challenge  Challenge  Challenge 

(x  £1 .85)  (1 ,85<x<2.05)  (x  ^2.05)  Gamma  =  -0.31 

N  =  18  N  =  12  N=  16  p=  0.05 


Relationship  to  project  performance: 

Moderately  strong  negative 

relationship 

Project  Challenge 


Gamma 

P 

-31% 

5.0% 

Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

22% 

28% 

50% 

1.85 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.85 

42% 

58% 

0% 

2.05 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.05 

38% 

38% 

25% 

4.0 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Reqts+Tech  Solution  with  Project  Challenge 


1.00H 


0.75 


0.5O 


0.25  H 


0.00  “I 


Lower  Challenge 

CPC  <  1 .95 


Lower 
REQ  +TS 
Capability 
(x  £  2 .3) 

N  =8 

Gamma  =  0.57 
p- value  =  0.02 


Moderate 
REQ+TS 
Capability 
C2.8<x<3.1) 
H  =  6 


Higher  Challenge 

CPC  >1.9) 


Higher 
REQ  +TS 
Capability 
(x  £  3. 1 ) 
H  =7 


Lower 
REQ+TS 
Capability 
(x  ^2.8) 
H  =7 


Moderate 
REQ  +TS 
Capability 
(2.3<x<3.1) 
N  =  7 


Higher 
REQ  +TS 
Capability 
(x  £  3.1) 
H  =11 


Gamma  =0.54 
p-value  =0.03 


Best 

Performance 
(x>  3.0) 

Moderate 
Performance 
(2 .5  <  x  <  3 .0) 

Lower 

Performance 
(x  <  2 .5) 


Relationship  to  project  performance:  Very  strong  positive  relationship 


SE  Capability  +  Project  Challenge 


REQ+TS+PC 


Gamma 

P 

63% 

0.0% 

Lower 

Min. 

^ r 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

67% 

33% 

0% 

1.7 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.7 

25% 

45% 

30% 

2.3 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.3 

14% 

36% 

50% 

4.0 
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STRENGTH  THHtKGH  INDUSTRY  &  TECHNOLOGY 


n 


Relating  Project  Performance  to 
Project  Challenge  and  SE  Capability 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Effectiveness 

Relationship  of  SEC  to  Performance 


Supplier  Systems  Engineering  Capability^ 

Relationship  to 

Project  Performance 

Relationship 

(Gamma^l) 

Section 

Reference 

Project  Planning 

Weak  positive  relationship 

+0.13 

5.1. 3.2 

Project  Monitoring  and  Control 

Weak  negative  relationship 

-0.13 

5.1.3.3 

Risk  Management 

Moderately  strong  positive 
relationship 

+0.28 

5.1. 3.4 

Requirements  Development  &  Management 

Moderately  strong  positive 
relationship 

+0.33 

5.1.3.5 

Trade  Studies 

Strong  positive  relationship 

+0.37 

5.1. 3.6 

Product  Architecture 

Moderately  strong  to  strong  positive 
relationship 

+0.40 

5.1.3.7 

Technical  Solution 

Moderately  strong  positive 
relationship 

+0.36 

5.1.3.8 

Product  Integration 

Weak  positive  relationship 

+0.21 

5.1. 3.9 

Verification 

Moderately  strong  positive 
relationship 

+0.25 

5.1.3.10 

Validation 

Moderately  strong  positive 
relationship 

+0.28 

5.1.3.11 

Configuration  Management 

Weak  positive  correlation 

+0.13 

5.1.3.12 

IPT-Related  Capability 

Moderately  strong  positive  correlation 

+0.34 

5.1.3. 1 

63 


STRENGTH  THROUGH  INDUSTRY  &  TCCHNOLOGY 


SE  Effectiveness 

Methodology  (In  Detail) 


SEEC  Activities 


NDIASED 
active  roster 


o 


NDIAmg’t  f 
input 


Identify 

industry 

members’ 

focals 


Company  Focal 

Activities 


Respondent 

Activities 


SEI  Activities 


Contact 

Provide 

Focal 

Focal 

focals,  brief 

Web 

contact  #1 

contact  #2 

the  survey 

-> 

access 

-► 

to 

-> 

to 

process, 

data  to 

expedite 

expedite 

solicit  support 

focals 

response 

response 

v _ y 

v _ y 

V _ _ J 

V _ . 

Report* 
findings  to 
NDIA  and 
OSD 


Identify 
respondents 
and  report 
number  to 
SEI 


Solicit 

respondents 
and  provide 
Web  site 
access  info 


Responde 
nt  contact 
#1  to 
expedite 
response 


Responde 
nt  contact 
#2  to 
expedite 
response 


Report 
number  of 
responses 
provided  to 
SEI 


Complete  questionnaire 
and  submit  to  SEI 


Collect  responses  and 
response  rate  data 


Report 
completion 
to  focal 


Analyze  data  and 
report  to  SEEC 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Effectiveness 

Methodology  (Mathematical  Statement) 


Perf=  f(PC,  PE,  SEC,  AC) 


Perf-  Project  Performance 
PC  -  Project  Challenge 
PE  -  Project  Environment  PE 
SEC  -  Systems  Engineering  Capability 
AC  -  Acquirer  Capability 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

Summary  of  Relationships 


Driving  Factor 

Relationship  to  Project 
Performance 

Description 

r 

Requirements  and 
Technical 

Solution  Combined 
with  Project  Challenge 

Very  strong  positive 

+0.63 

Combined 

Requirements  and 
Technical  Solution 

Strong  positive 

+0.49 

Product  Architecture 

Moderately  strong 
to  strong  positive 

+0.40 

Trade  Studies 

Moderately  strong 
to  strong  positive 

+0.37 

IPT-Related  Capability 

Moderately  strong 
positive 

+0.34 

Technical  Solution 

Moderately  strong 
positive 

+0.36 

Requirements 
Development 
and  Management 

Moderately  strong 
positive 

+0.33 

Driving  Factor 

Relationship  to  Project 
Performance 

Description 

r 

Total  Systems 
Engineering  Capability 

Moderately  strong 
positive 

+0.32 

Project  Challenge 

Moderately  strong 
negative 

-0.31 

Validation 

Moderately  strong 
positive 

+0.28 

Risk  Management 

Moderately  strong 
positive 

+0.28 

Verification 

Moderately  strong 
positive 

+0.25 

Product  Integration 

Weak  positive 

+0.21 

Project  Planning 

Weak  positive 

+0.13 

Configuration 

Management 

Weak  positive 

+0.13 

Process  Improvement 

Weak  positive 

+0.05 

Project  Monitoring  and 
Control 

Weak  negative 

-0.13 
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EXPERIENCE.  RESULTS. 


Agenda 

•  Background 

•  System  Management  Overview 

•  Case  Study:  Insight 

•  The  Benefits  of  Insight 
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The  Aegis  Weapons  System 

•  Aegis  is  the  world’s  premier  naval  surface 
defense  system.  It  is  capable  of 
simultaneous  operation  defending  against 
air,  surface,  subsurface  and  ballistic  missile 
threats. 


•  There  have  been  7  major  releases,  known 
as  baselines,  in  the  nearly  4  decade  history 
of  Aegis. 


•  Recent  baselines  have  focused  on  re¬ 
engineering  the  weapons  system  to  take 
advantage  of  commercially  available  off- 
the-shelf  (COTS)  operating  environments 
(OE). 


EXPERIENCE.  RESULTS 
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A  Migration  to  Open  Architecture 


Proprietary  Systems 


Manufactured  hardware  and 
developed  software 


Open  Technology 


Emphasis  on  COTS  hardware 
and  software  integration 


Computing  System  Management  functions  have  become  more  complex 

with  the  adoption  of  COTS  technology 
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Open  Technology 


#  concurrent 


Intel.  jSa  ^|l  redhat  XOHl 

invent 


•  An  Aegis  ship  not  much  different  from  a  large-scale 
commercial  data  center. 


•  The  weapons  system  is  comprised  of  a  standard  operating 
environment  with  unique  components  not  seen  in 
commercial  architectures. 
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An  Integrated  Computing  Environment 


Air, 

Missile  & 
National 
Defense 


EXPERIENCE.  RESULTS. 


Networked  Computer  Systems 


System  Management  Overview 


Application  Management 

•  Manage  where  applications  are 
running 

•  Manage  runtime  state  of  the 
applications 

•  Manage  recovery  and 
reconfiguration 

•  Assess  health  status  of  the 
applications 


Equipment  Management 

•  Node/Server  Management 

-  Diagnostics 

-  Performance  Monitoring 

•  Network  Management 

•  Asset  Management 

-  Validation  and  Verification 

-  Software  Distribution 


Fault  Detection  /  Fault  Isolation 
Root  Cause  Analysis 
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Aegis  System  Management 


In  the  current  baselines, 
System  Management  for  the 
Aegis  Weapons  System  is 
comprised  of  multiple 
components. 


Each  component  exists  as  a 
standalone  entity. 


Together,  they  interoperate  to 
provide  an  end-to-end  solution 
for  Operational  Readiness 
Assessment  of  the  combat 
system. 


ami 

iJ. 


Insight  is  a  key  component  for  managing  the  operating  environment  of  the 

Aegis  Weapons  System 
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Case  Study:  Insight 


EXPERIENCE.  RESULTS. 
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What  is  Insight? 

•  An  integrated  computing  system  management  toolkit. 

•  It  is  a  highly  configurable  suite  of  open  source, 
commercially  available,  and  developed  tools  that  perform 
system  management  functions  across  the  enterprise. 

-Hardware  and  software  diagnostics 

-Performance  monitoring 

-  Network  management 

-  System  control 
-Validation  and  verification 
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The  Catalyst  For  Insight 


1994 

Baseline  6  signals  the 
emergence  of  COTS 
processors  to  augment  the 
legacy  computer  suite. 


2000 

Deployment  of  Baseline  6  is 
plagued  by  schedule  delays. 
Ad-hoc  tools  with  weak  fault 
detection  and  inefficient  fault 
isolation. 


2002 

Initial  version  of  Insight 
deployed  to  local  data  centers. 


1969 

Aegis  begins.  Baselines  1 
through  5  are  deployed  with  a 
proprietary  operating 
environment. 


1999 

Baseline  7  signals  the 
emergence  of  an  all-COTS 
computer  suite. 


2001 

CSC  produced  a  White  Paper 
to  address  the  need  for  a  well 
defined  system  management 
strategy  on  the  COTS 
baselines.  Design  and 
development  of  Insight 
begins. 


2003  -  Current 

Insight  is  deployed  to 
Baseline  7  for  use  in  shipyard 
integration  activities  and 
shipboard  Operational 
Readiness  Assessment. 
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Insight  Component  Architecture 

•  Framework 

-  Graphical  User  Interface, 

Network  Services,  and  Remote 
Agents 

•  Tools 

-  Configurable  collection  of  “best- 
of-breed”  products  and  utilities 
to  perform  system  management 
functions 

•  Configuration  Data 

-  Easily  tailors  Insight  to  the 
needs  of  the  open  architecture 
enterprise  being  managed 

•  Baseline  Data 

-  Repository  of  expected  OE  configuration  state  information 
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Extensive  Use  Of  Open  Source  Technology 


•  Over  40%  of  Insight  is 
comprised  of  Open  Source 
software 

-  Permits  selection  of  cost 
effective,  best-of-breed 
solutions 

-  Reduces  development  time 

-Leverages  intellectual 
resources  from  the  world 
wide  development 
community 


Functional  Composition 


Capitalizing  on  Open  Source  saved 
approximately  $1M  in  development  costs 
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Application  Management 


Equipment  Management 


•  Manage  where  applications  are 
running 

•  Manage  runtime  state  of  the 
applications 

•  Manage  recovery  and 
reconfiguration 

•  Assess  health  status  of  t 
applications 


Insight’s  Toolkit 


•  Node/Server  Management 

-  Diagnostics 

-  Performance  Monitoring 

•  Network  Management 

•  Asset  Management 

-  Validation  and  Verification 

-  Software  Distribution 


Fault  Detection  /  Fault  Isolation 
Root  Cause  Analysis 
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Insight  Functionality 


On-demand  Testing 
Runtime  Monitoring 
Interactive  Control 


^^Diagnostics 
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Consistent  Interface  for  Disparate  Tools 

•  Standardizes  test  execution 

•  Hides  tool  variability  from  the  user 

•  Provides  common  results 


Insight  standardizes  the  execution  and  results  of  each  tool, 
regardless  of  platform  or  tool  origin 
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Hardware  and  Software  Diagnostics 

•  Fault  Detection  /  Fault  Isolation  (FD/FI) 

•  Assess  the  operational  state  of  hardware  and  software 
components 

-  NTDS  Devices 

-  I/O  Boards 

-Specialized  Processors 
-Tactical  Interfaces 
-Middleware  Components 
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Increased  Productivity 


Integration  Productivity  Rate 


The  availability  of  Insight  has  increased 
the  productivity  rate  to  over  90%  by 
providing  rapid  identification  of  operating 
environment  issues. 


100% 

80% 

60% 

40% 

20% 

0% 


2001  2002  2003 


Rapid  Resolution  =  Increased  Productivity  =  Significant  Savings 
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Performance  Monitoring 

•  Assess  infrastructure  performance  under  operational 
conditions 

-CPU  utilization 

-Memory  utilization 

-  Process  activities 
-Disk  usage 

-  I/O  activity 

-  Kernel-level  trace 

•  Data  is  represented  in  real-time  as  well  as  archived  for 
statistical  representation 
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Insight  assesses  operational 
readiness  of  combat  system 
components  from  a  single 
console 


Validation  and  Verification 

•  System  Operability  Test 

•  Checks  a  number  of  Operating  Environment  configuration 
parameters  to  determine  overall  system  operability 

-Configured  Devices 

-Filesystem  Integrity 

-Kernel  Tunables 

-  Network  Tunables 

-Installed  Packages 

-OS  Version  and  Patches 

-Firmware 

•  Ensures  that  the  current  state  of  a  host’s  OE  is  consistent  with 
established  baseline  data 
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Automated  Validation  of  the  Operating  Environment 


Integration  and  Test  Facility 


Deployed  System 


Insight  ensures  that  the  deployed  configuration 
is  identical  to  the  staged  environment 
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EXPERIENCE.  RESULTS. 


Automated  System  Validation:  Months  to  Minutes 


There  are  over  6000  1 

1^^  - 

configuration 

parameters  on  each 

node  that  are  optimized 

for  the  application 

Manual  validation  of 
the  configuration  for 
the  entire  computer 
suite  would  take 
nearly  4  staff  months 


r-  HH 

mil  -  if  i 

1  **  1  ^  1 

zjaoiE] 

saaii. 
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Insight  provides 
accurate  validation  for 
the  entire  computer 
suite  within  3  minutes 


1 


Automated  validation  provides  complete  confidence  in  the  deployed 
configuration  and  saves  significant  taxpayer  dollars 
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Network  Management 

•  Assess  health  status  and  performance  of  the  local  area 
network 

-LAN  operability 

-  Routes 

-  Network  utilization 


System  Control 

•  Manage  the  run-time  state  of  the  tactical  applications 
-System  Initialization 
-Shutdown 
-  Reboot 
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Run-time  state  of  the  system 
can  be  monitored  and 
controlled  from  a  single  console 
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Concept  of  Operations 

•  Insight  is  used  throughout  the  life-cycle  of  the  combat 
system. 


Development  Data 
Center 


Validation  of  the  build 
environment 


Staging  and  Test 
Facility 


Shipyard  Integration 


System  validation,  diagnostics,  operability  tests 


Deployed  Systems 


Runtime  status 
monitoring,  operability 
tests,  diagnostics 


Insight  has  become  standard  operating  procedure  for  Aegis 
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Safety  Checks 

Insight’s  safety  agent  prevents 
intrusive  functions  during  tactical 
operations. 


The  Benefits  of  Insight 


A  Vital  Maintenance  Solution  for 
the  Aegis  Weapons  System 


I 


Proprietary  systems  had  an 
available  toolset. 


Insight  provides  a 
robust  toolset  for  COTS 
systems. 
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Applying  Insight  to  Support  Various  Activities 


Insight  is  a  key  component  for  Aegis  System  Management 
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Enhanced  Operational  Readiness 


Dispatching  engineers  to 
troubleshoot  shipboard 
issues: 

$75K 


Delaying  a  missile  test  due  to 
system  instability: 

$1M 


Meeting  deployment  schedules 
and  increasing  credibility  with 
your  customer: 

Priceless 


Insight  is  playing  a  vital  role  ensuring  that  ships  are  deployed  on  schedule 
for  immediate  use  in  various  theatres  of  operation 
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Innovation:  Delivered 
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*  Deployed  for  3  V2  years 

*  Multiple  Baselines 

*  Domestic  and  Foreign  Test  Sites 

*  Domestic  and  Foreign  War  Ships 

*  Certified  for  Tactical  Operations 
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Update: 

OSD  Systems  Engineering 
Revitalization  Efforts 

23  October  2007 


Col  Rich  Hoeferkamp 
Ms.  Sharon  Vannucci 

Systems  and  Software  Engineering 
(Enterprise  Development) 

Office  of  the  Deputy  Under  Secretary  of  Defense  (Acquisition  &  Technology) 


General  Outline 


>  What’s  happening  in: 

•  Policy 

•  Education  &  Training 

•  Guidance 

>  Other  Initiatives 


>  Topics  for  Discussion 


System  Engineering  Policies 


All  programs  shall  develop  a  SE 
Plan  (SEP) 

Each  PEO  shall  have  a  lead  or  chief 
systems  engineer  who  monitors  SE 
implementation  within  program 
portfolio 

Event-driven  technical  reviews  with 
entry  criteria  and  independent 
subject  matter  expert  participation 

OSD  shall  review  program’s  SEP  for 
major  acquisition  programs  (ACAT 
ID  and  1AM) 


Technical 

Planning 


Technical 

Leadership 


Technical 

Execution 


Technical 

Excellence 


Technical  planning  upfront  and  early 


New  SE  Policy  in  Draft  DoDI  5000.2, 
Operation  of  the  Defense  Acquisition  System 


3.0  PROCEDURES 

3.1.  Defense  Acquisition  Management  Framework.  Figure  1  depicts  the 
Defense  Acquisition  Management  Framework. 


•  Process  entry  at  Milestone  A,  B,  or  C 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 


A  A! 


(Program 

Initiation) 


A 


IOC 


FOC 


1  Concept 
Refinement 

Technology 

Development 

System  Development 
&  Demorj^tratjon 

Production  & 
Deployment 

Operations  & 
Support 

\  Concept 
/  Decision 

f  A  Critical  \ 
I  <  >  Design  ) 

V  V  Review  J 

LRIP/IOT&E  A  Decision 

V  Review 

Pre-Systems  Acquisition 


ferns  Acquisition 


Sustainment 
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New  SE  Policy  in  Draft  DoDI  5000.2 


§  3.6.5.  “systems  engineering  planning  shall  support”  the  Technology 
Development  phase 

§  3.7.8.  “System  Design  [first  phase  of  SDD]  shall  include  the 

establishment  of  the  functional,  allocated,  and  product  baselines  for  all 
configuration  items.  The  CDD  [Capability  Development  Document]  and 
Systems  Engineering  Plan  [SEP]  shall  guide  this  effort.” 

§  3.7.9.  “Critical  Design  Review  (CDR).  The  system-level  CDR  provides 
an  opportunity  to  assess  design  maturity  as  evidenced  by  measures 
such  as  successful  completion  of  subsystem  CDRs;  the  percentage  of 
hardware  and  software  product  build-to  specifications  and  drawings 
completed  and  under  configuration  management;  planned  corrective 
actions  to  hardware/software  deficiencies;  adequate  developmental 
testing;  an  assessment  of  environment,  safety  and  occupational  health 
risks;  a  completed  failure  modes  and  effects  analysis;  the  identification 
of  key  system  characteristics  and  critical  manufacturing  processes;  and 
an  estimate  of  system  reliability  based  on  demonstrated  reliability  rates, 
etc.” 
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New  SE  Policy  in  Draft  DoDI  5000.2 


§  3.7.9. 1 .  “The  PM  shall  provide  a  CDR  Report  to  the  MDA  that  provides 
an  overall  assessment  of  design  maturity  and  a  summary  of  the  system- 
level  CDR  results  which  shall  include,  but  not  be  limited  to: 

§  3.7.9. 1 .1 .  The  names,  organizations,  and  areas  of  expertise  of 
independent  subject  matter  expert  participants  and  CDR  chair; 

§  3.7.9. 1 .2.  A  description  of  the  product  baseline  for  the  system  and  the 
percentage  of  build-to  packages  completed  for  this  baseline; 

§  3.7.9. 1 .3.  A  summary  of  the  issues  and  actions  identified  at  the  review 
together  with  their  closure  plans; 

§  3.7.9. 1 .4.  An  assessment  of  risk  by  the  participants  against  the  exit 
criteria  for  the  SDD  phase;  and 

§  3.7.9. 1 .5.  Identification  of  those  issues/risks  that  could  result  in  a 
breach  to  the  program  baseline  or  substantively  impact  cost,  schedule  or 
performance.” 

§  3. 7.9.2.  “The  MDA  shall  review  the  CDR  Report  and  the  PM’s  resolution/ 
mitigation  plans  and  determine  whether  additional  action  is  necessary  to 
satisfy  SDD  phase  exit  criteria  and  to  achieve  the  program  outcomes 
specified  in  the  APB.” 
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New  SE  Policy  in  Draft  DoDI  5000.2 


§  3.10.6.  “Program  Support  Review  (PSR).  The  Office  of  the  USD(AT&L)/ 

Systems  and  Software  Engineering  shall  conduct  PSRs  for  MDAPs  to  assess 
the  application  of  technical  planning  and  management  processes  and  assist 
the  program  office  in  identifying  and  mitigating  cost,  schedule,  and 
performance  risk.  PSRs  shall  be  conducted  prior  to  each  milestone  event, 
before  approval  of  the  SDD  acquisition  strategy,  and  at  other  times  as  directed 
by  the  USD(AT&L).  The  results  of  a  PSR  shall  inform  the  OIPT  and  be 
provided  to  the  MDA.  PSRs  on  MAIS  programs  shall  be  conducted  when 
requested  by  the  MDA.” 

Enclosure  3.  Table  E3.T2:  SEP  mandated  at  milestones  A,  B,  and  C 

Enclosure  5.  §  E5.7.2.  “The  Office  of  the  USD(AT&L)/  Systems  and  Software 
Engineering  shall  conduct  an  independent  Assessment  of  Operational 
Test  Readiness  (AOTR)  for  all  ACAT  ID  and  special  interest  programs 
designated  by  the  USD(AT&L).  Each  AOTR  shall  consider  the  risks 
associated  with  the  system’s  ability  to  meet  operational  suitability  and 
effectiveness  goals.  This  assessment  shall  be  based  on  capabilities 
demonstrated  in  DT&E  and  OAs  and  criteria  described  in  the  TEMP.  The 
AOTR  report  shall  be  provided  to  the  USD(AT&L),  DOT&E,  and  CAE.” 

§  E5.7.3.  “The  CAE  shall  consider  the  results  of  the  AOTR  prior  to 
making  a  determination  of  materiel  system  readiness  for  IOT&E.” 
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New  SE  Policy  in  Draft  DoDI  5000.2 


Enclosure  12.  Systems  Engineering. 

E12.1.  Systems  Engineering  Across  the  Acquisition  Lifecycle. 


E12.2.  Systems  Engineering  Plan  (SEP). 

El  2.2.1.  “Program  managers  shall  prepare  a  SEP  for  each  milestone  review, 
beginning  with  A.” 

El  2.2.2.  “The  MDA  shall  be  the  approval  authority  for  the  SEP.” 

El 2.3.  Systems  Engineering  Leadership.  Each  Program  Executive  Officer  shall  have 
a  lead  or  chief  systems  engineer  on  his  or  her  staff  responsible  to  the  PEO  for 
systems  engineering  across  the  PEO’s  portfolio  of  programs  and  shall: 

El 2.3.1 .  Review  assigned  programs’  SEPs  and  oversee  their  implementation. 

El  2.3.2.  Assess  performance  of  subordinate  lead  or  chief  system  engineers. 

El 2.4.  Technical  Reviews.  Technical  reviews  shall  be  event  driven,  conducted  when 
documented  entrance  criteria  are  met,  and  include  participation  by  subject  matter 
experts  who  are  independent  of  the  program. 

El 2.5.  Configuration  Management.  Documented  in  the  SEP,  the  configuration 
management  approach  shall  identify,  document,  audit,  and  control  the  functional 
and  physical  characteristics  of  the  system  design,  track  any  changes,  and  provide 
an  audit  trail  of  program  design  decisions  and  design  modifications. 
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New  SE  Policy  in  Draft  DoDI  5000.2 


E12.6.  Environment,  Safety,  and  Occupational  Health  (ESOH).  The  PM  shall  use 
the  methodology  in  MIL-STD-882D  to  assess  ESOH  risk,  eliminate  ESOH 
hazards  where  possible,  manage  the  risks  that  cannot  be  eliminated,  and  report 
on  the  status  of  ESOH  risk  at  technical  reviews. 

E12.6.1.  Programmatic  ESOH  Evaluation  (PESHE).  The  PM  for  all 
programs,  regardless  of  ACAT  level,  shall  prepare  a  PESHE  and  summarize  it  in 
the  acquisition  strategy. 

El 2.5.2.  NEPA/EO  12114.  The  PM  shall  conduct  and  document  NEPA/EO 
12114  analyses,  to  be  approved  by  the  CAE,  for  which  the  PM  is  the  action 
proponent. 

El 2.6.3.  Mishap  Investigation  Support.  The  PM  will  support  system-related 
Class  A  and  B  mishap  investigations. 

El 2.7.  Corrosion  Prevention  and  Control.  Each  ACAT  I  program  shall  document  its 
strategy  in  a  Corrosion  Prevention  Control  Plan  at  Milestones  B  and  C. 

El 2.8.  Modular  Open  Systems  Approach  (MOSA).  Program  managers  shall 
employ  MOSA. 

E12.9.  Data  Management  and  Technical  Data  Rights.  Program  managers  for  all 
ACAT  I  and  II  programs  shall  assess  their  long-term  technical  data  needs  and 
document  them  in  a  Data  Management  Strategy  which  shall  be  approved  in  the 
context  of  the  acquisition  strategy  prior  to  issuing  a  contract  solicitation. 
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Education  &  Training 


>  What’s  available 

•  On-line  Continuous  Learning  Modules  (CLMs): 

-  Reliability  and  Maintainability 

-  Technical  Reviews 

-  Technical  Planning 

-  MOSA  (new) 

-  Trade  Studies  (new) 

•  On-line  introductory  course  SYS  101 

•  On-line  intermediate  course  SYS  202 

•  Intermediate  classroom  course  SYS  203 

•  Advanced  classroom  course  SYS  302 

•  New  “SPRDE/Program  Systems  Engineer”  track 

•  “Core  Plus”  career  guidance  (new) 

>  What’s  Coming 

•  Data  Management  CLM 

•  ECPCLM 


New  SPRDE/PSE  &  SE  Career  Path 
Certification  Criteria 


Education  &  Training 
New  SPRDE  Key  Tenets 


>Personnel\  those  currently  certified  SPRDE-SE  retain 
their  certification 

>Positions:  those  currently  coded  as  “S”  (SPRDE-SE) 
retain  that  designation — subsequently,  Components  shall 
review  positions  to  determine  if  they  should  remain  coded 
as  SPRDE-SE  (“S”)  or  if  they  should  be  recoded  as 
SPRDE-PSE  (“Code  TBD”) 


Guidance 


•  What  s  available: 

-  Systems  Engineering  Plan  (SEP)  Preparation  Guide,  V2  (just 
released) 

-  Risk  Management  Guide  for  DoD  Acquisition 

-  DoD  Guide  for  Achieving  Reliability,  Availability,  and  Maintainability 

-  Integrated  Master  Plan/Integrated  Master  Schedule  (IMP/IMS)  Guide 

-  Guide  to  Integrating  SE  into  DoD  Acquisition  Contracts 

-  Understanding  and  Leveraging  a  Supplier’s  CMMI  Efforts:  A 
Guidebook  for  Acquirers 

-  Risk  Assessment  Technical  Review  Checklists  (new) 

•  What’s  coming: 

-  Systems  of  Systems  SE  Guide 

-  Update  to  Defense  Acquisition  Guidebook 

-  Chapter  4  --  Systems  Engineering 

-  Chapter  9  --  Test  and  Evaluation 


Updated  SEP  Prep  Guide 


>  Includes  sections  by  program  phase: 

•  MS  A/Technology  Development 

•  MS  B/System  Development  &  Demonstration 

•  MS  C/Production  &  Deployment  and  Operations  &  Support 

>  Each  section  provides  more  “food  for  thought”  relative  to 

the  technical  planning  focus  areas  for  that  phase 

•  Program  Requirements 

•  Technical  Staffing 

•  Technical  Baseline  Management 

•  Technical  Review  Planning 

•  Integration  with  Overall  Management  of  the  Program 


Vision:  SE  Plan  Unification 


•  Acquirer/Supplier-developed  technical  plan  for  SE  implementation 

•  Acquirer/Supplier  shared  roles  and  responsibilities  in  SE  effort 

•  Acquirer/Supplier  conducted  event  driven  technical  reviews 

•  Acquirer/Supplier  teaming  on  linkage  with  other  program  plans 


TRR  Checklist 


D  I  E  |  F  |  B  |  H  |  I . 


Systems  Engineering  for  Mission  Success’ 


Test  Readiness  Review 

Program  Risk  Assessment  Checklist  [17  May  2007  version) 

OVERVIEW:  Although  the  checklist  can  be  printed  and  completed  as  a  "hard  copy",  it  is  designed  to  be  completed  electronically  as  an 
Excel  spreadsheet.  When  viewed  electronically,  the  small  number  buttons  in  the  upper  left  corner  of  the  screen  are  used  to  select  the 
level  of  indenture  for  the  questions  in  the  checklist.  A  left  mouse  click  on  a  number  button  will  expand  or  collapse  the  entire  checklist  to 
the  desired  level.  A  left  click  on  the  "+"  symbol  in  the  left  margin  of  the  spreadsheet  will  expand  the  level  of  indenture  for  that  section.  A 
left  click  on  the  symbol  in  the  left  margin  of  the  spreadsheet  will  collapse  the  level  of  indenture  for  that  section.  The  buttons  in  Row  11 
run  specific  macros.  The  buttons  in  Column  A  allow  a  user  to  designate  and  sort  specific  questions  as  "Special  Interest"  (i.e..  High 
Priority,  Flagged,  Question).  The  colored  buttons  in  Row  11,  Column  C  allow  the  user  to  sort  questions  by  Technical  Discipline,  to 
provide  a  Level  1  roll-up  of  the  risk  characters  assigned,  or  to  hide  specific  information.  For  example  selecting  the  "Logistics"  button 
results  in  the  display  of  all  Level  1  Logistics-related  questions  and  assigned  information.  All  other  questions  will  be 
hidden. 

COMPLETING  THE  CHECKLIST: 

1.  In  the  upper  right  corner  of  the  checklist,  enter  the  name  of  the  program  being  reviewed,  the  date(s)  of  the  review,  along  with  the  name, 
technical  specialty  of  the  person(s)  completing  the  checklist'. 

2.  A  "Risk  Character"  (i.e.,  R  \  Y  /  G  /  U  /  NA)  should  be  assigned  for  each  question  by  direct  entry  or  left  clicking  in  each  box  to  activate  ths 
down"  menu.  The  assigned  Risk  Characters  will  automatically  total  and  display  in  the  Level  1  (and  Level  2,  as  applicable)  row(s).  Selectit 
summary  tab  (Excel  "Sheet")  at  the  bottom  of  the  checklist  will  provide  a  summary  of  all  questions  assigned  a  particular  risk  character  (e.t 
the  RED  tab  will  display  all  questions  assigned  a  RED  risk  character).  To  delete  a  "Risk  Character"  from  a  box,  in  the  box  and  press  the  "[ 
button  on  the  keyboard. 

3.  Any  question  requiring  further  attention  (Special  Interest)  should  be  similarly  marked  in  Column  A  as  "High  Priority",  "Flagged",  or  "Ql 
facilitate  follow-up. 


Name  of  the  program  being  reviewed  /  date 


Name  /  Code  /  Technical  Specialty  of  reviewer 


C AUTI  ON:  Entries,  oiienges,  ds/stions  or  comments  shouid  oniy  be  modo  on  the  cJieckiist  Ary  entrres  entered 
dree  fin  on  the  summery peges  m/f  no  bo  recorded  within  the  checklist  end  wii  disebie  iinkege  between  the  checklist 


SAVING  THE  CHECKLIST:  Save  the  completed  checklist  in  a  new  file  with  a  unique  name  such  as  "UAV  TRR  8Feb07ajo". 
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TRR  Checklist 


Filter  Mode  MUM 
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Corrosion 


>  Overarching  strategy:  transcend  traditional  control  methods, 
organizations,  management  and  funding  approaches 

>  Attack  corrosion  early  in  acquisition  or  construction 

>  Focus  life-cycle  corrosion  research  and  development  efforts 
on  four  primary  areas 

•  Materials  and  manufacturing  processes  that  prevent  or  reduce 
the  incidence  and  effects  of  corrosion 

•  Detection  of  corrosion  in  fielded  systems  and  facilities  and 
prognosis  of  the  expected  growth,  potential  impact  and  predicted 
effects 

•  Coatings,  treatments  and  other  applications  to  prevent,  arrest  or 
retard  corrosion 

•  Repair  processes  that  restore  materials  to  an  acceptable  level  of 
structural  integrity  and  functionality 

>  Publish  direction  and  guidance  regarding  corrosion  prevention 
and  mitigation  policies  and  strategies  at  all  DoD  and  Service 
levels 
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SE  Research 


>  Recognizes  need  to  advance  the  practice  of  systems  engineering  within  DoD 

•  Conduct  innovative  research  into  new  SE  methods,  processes,  and  tools  (MPTs)  to 
address  recurring  problems  in  the  acquisition  of  systems  and  services. 


>  Currently  investigating  stand-up  of  an  SE  research  University  Affiliated  Research  Center 
(SER  UARC)  to  help  accomplish  this 

>  SE  UARC  mission:  to  research  and  analyze  advanced  and  emerging  systems 
engineering  practices  and  relevant  technologies  to  address  the  full  spectrum  of  DoD 
systems  across  the  Department, 

•  From  capability  areas,  enterprise  systems,  systems  of  systems,  and  interoperability 
down  to  subsystems  and  configuration  items 

•  Goal  is  to  ensure  consistency  and  SE  excellence  throughout  acquisition  life  cycle 

>  Bottom  line: 

•  We’ve  made  great  strides,  but  much  still  to  do  to  change  root  cause  behaviors  on 
programs. 

•  SE  research  will: 

-  Inform  the  current  state  of  SE  practice  on  programs 

-  Provide  a  means  to  explore  and  exploit  concepts  to  enable  design  and  development  of 
complex  systems 

-  Underpin  effective  integration  of  program/business  processes  with  technical  management 
MPTs  throughout  the  acquisition  life  cycle. 


Topics  for  Discussion 


>  What  are  your  thoughts  on: 

•  The  “help”  you  are  getting  from  OSD  (policy,  guidance,  Education  & 
Training,  Programs  Support  Reviews,  SEP  reviews,  etc.) 

-  Have  the  SE  policy  memos  of  2004  caused  you  to  do  anything 
different? 

•  What  it  takes  to  deploy  effective  SE  across  all  programs 

•  The  availability  of  resources  you  have  to  put  on  SE,  and  ability  to  train 
them 

•  The  need  for  independent  chair  at  technical  reviews  outside  of  the 
program 

•  Mandating  development  of  the  SEP  prior  to  RFP  release 

•  A  unified  acquirer/supplier  SEP 


Topics  for  Discussion 


>Other  questions  to  consider: 

•  How  are  technical  reviews  conducted  and  when  are  they  held?  Are 
they  schedule  driven  or  event-based? 

•  Do  you  have  “technical  baselines”?  What  process  do  you  use  to  track 
and  manage  them? 

•  Who  does  your  planning  (e.g.  writes  your  SEP  and  Risk  Plan)?  Are 
these  plans  used  and  are  they  value  added? 

•  How  do  the  SEP,  Risk  Plan,  TEMP  and  other  technical  documents 
integrate  with  the  acquisition  strategy? 


Way  ahead 


Taking  SE  to  the  next  level  -  Expand  institutionalization. .. 

>  Change  culture,  both  vertically  and  horizontally,  across 
Government  and  DoD  contractor  workforce 

•  Expand  outreach  effort  to  the  product  centers,  program 
offices,  and  key  industry  partners 

•  Continue  to  leverage  collaboration  with  DAU  and 
academia  to  further  develop  the  workforce 


SE  links 


Systems  and  Software  Engineering 
(updated  website): 

http://www.acg.osd.mil/sse 


DAU: 

http:www.dau.mil/basedocs/traininqcourses.asp 


OUSD  (AT&L)  Organization 


Defense  Procurement 
and  Acquisition  Policy 


Industrial 

Programs 


Small  Business 
Programs 


Defense  Contract 
Management  Agency 


Systems  and  Software  Engineering 

Organizational  Core  Competencies 


CORE  COMPETENCIES 


•  SE  Policy 

•  SE  Guidance 

•  SE  in  Defense  Acquisition 
Guidebook 

•  Technical  Planning 

•  Risk  Management 

•  Reliability/Maintainability 

•  Integrating  SE  into 
Systems  Acq  contracting 

•  SoS  SE  Guide 

•  SE  Education  and  Training 

•  DAU  SE  Curriculum 

•  SPRDE  Certification  Rqmt 

•  Corrosion 

•  R-TOC 

•  Value  Engineering 


CORE  COMPETENCIES 


DT&E  Policy 
DT&E  Guidance 

•  T&E  in  Defense 
Acquisition  Guidebook 

•  TEMP  Development 
Process 

DT&E  Education  and 
Training 

•  DAU  DT&E  Curriculum 

•  DT&E  Certification  Rqmt 
Joint  Testing,  Capabilities 
&  Infrastructure 
Targets  Oversight 

Acq  Modeling  &  Simulation 
Energy 

DSOC/Acq  Tech  Task  Force 


CORE  COMPETENCIES 


SWE  and  SA  Policy 
SWE  and  SA  Guidance 

*  SoS,  SA  Guides 

SWE  and  SA  Education  and 
Training 

*  DAU  SW  Acq  Curriculum 

*  Continuous  Learning 
Modules  for  SWE,  SoS,  SA 

Software  Engineering 

*  Acquisition  Support 

*  Software  Engineering 
Institute  (SEI) 

Process  Improvement 

*  CMMI  Sponsor 
DoD/National  Software 
Investment  Strategy 


CORE  COMPETENCIES 


Support  of  ACAT  I  and 
Other  Special  Interest 
Programs  (MDAP,  MAIS) 
Assessment  Methodology 
(Program  Support 
Reviews  -  PSRs) 

T&E  Oversight  and 
Assessment  of  Operational 
Test  Readiness  (AOTR) 
Systems  Engineering  and 
Developmental  Test 
Planning  and  Support 
Lean/6-Sigma  Training/Cert 


Acquisition  program  excellence  through  sound  systems  and  software  engineering 


Potential  SE  Research  Focus  Areas 


>  Better  integration  of  SE  processes,  methodologies,  and  tools 

•  Consideration  of  toolsets  to  full  system  life  cycle  (CONOPS  analysis  through 
sustainment) 

•  Extensibility  of  modeling  languages  and  tools  to  SE  and  program  artifacts 

•  Technical  Baseline  management  tools 

•  Data  standards  alignment  with  other  domains  (logistics,  test,  ops  analysis) 

•  Enabling  EVM,  Risk  Management,  cost  estimating,  etc 

>  Alignment  of  systems  engineering  and  software  standards  (15288,  12207,  632, 
1220,  etc) 

>  SE  considerations  in  complex  Systems  of  Systems  environment 

•  Large  systems,  distributed  systems,  software  intensive  systems,  net-centric 
ops 

>  Enterprise-wide  SE  considerations 

>  Linking  of  architectures  with  SE  products/technical  baselines 


Potential  SE  Research  Focus  Areas 


>SE’s  Return  on  Investment  (ROI) 

•  Relationship  of  investment  in  SE  effort  to  reduction  in  life  cycle 
costs/program  predictability 

•  Metrics  that  support  technical  decisions  and  ROI 

>  Knowledge  management  SE  repositories 

•  Mechanisms  to  advance  and  share  the  state-of-the-practice 

>  Lean  Six  Sigma  opportunities  on  SE  processes 

>  SE  application  on  services  contracts 

>  Graduate/Continuing  Education  SE  education  needs 

>  SE  for  governance  and  strategic  choices  (investment) 

•  Advanced  training  concepts 


10th  Annual  Systems  Engineering  (SE)  Conference 


STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


The  Effectiveness  of  Systems  Engineering: 
On  Federal  System  Development  Programs 


First  Public  Release 

Of  Major  New  NDIA  Study  by 
The  Systems  Engineering  Effectiveness  Committee 

(SEEC) 


Al  Mink 

SEEC  Member 
&  SRA  International 
October  2007 


STRENGTH  THHOIGH  INDUSTRY  &  TECHNOLOGY 


SE  Effectiveness 

Overview 


The  SE  Effectiveness  Survey 

Quantifies  the  relationship  between  the 
application  of  Systems  Engineering  best  practices 
and  the  performance  of  system  development  projects 


Systems  Engineering 
capabilities  deliver 
better  Project 


TODAY’S  OUTLINE 

1.  The  Challenge 

2.  The  Rigor 

3.  The  Results! 

4.  Conclusions  &  Caveats 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


The  Challenge 

Stakeholder  Analysis 


Those  interested  in  such  a  study  -  and  their  interests 


Customers 

•  DoD  #1  SE  Issue  -  “Inconsistent  SE  Practices  across  life  cycle” 

•  Validate  initiatives  to  revitalize  SE 

•  Increase  emphasis  of  SE  content  in  RFPs  and  Proposals 
Industry  (System  Developers  &  Integrators) 

•  Proposal  may  skimp  on  SE;  Budget  pressures  on  SE 
Associations  &  Academia 

•  Unable  to  fully  satisfy  their  members  and  students 
SE  professionals 

•  Lack  rigorous  justification  for  their  recommendations 
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STRENGTH  THHOIGH  JRDLSTRY  &  TECHNOLOGY 


The  Challenge 

Previous  Studies  -  Partial  Insights 


Gruhl,  National  Avionics  and  Space  Administration  (NASA),  1992 

Compared  upfront  expenditures  to  eventual  cost  growth 

Herbsleb,  Software  Engineering  Institute  (SEI),  1994 

Studied  ROI  on  process  improvement  in  software 

Honour,  International  Council  on  Systems  Engineering  (INCOSE), 
2002 

Surveyed  industry  to  compare  SE  Effort  to  cost  &  schedule 

Valerdi  &  Boehm,  Constructive  System  Engineering  Cost  Model 
(COSYSMO),  2004 

Developed  parametric  estimation  model  similar  to  COCOMO 

Boehm  &  Valerdi,  SE  ROI  (COCOMO),  2006 

Analyzed  SE  activities  from  COCOMO  II 

Others... 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


The  Challenge 

Previous  Studies  -  Summary 


STUDY 

APPLICABILITY 

Author  & 
Background 

Findings 

SE  Activities 

Definition  of 
Success 

Characteristics 
of  Project 

Gruhl  (1992) 

32  NASA  Pgms 

8-15%  Upfront 
Best 

First  two  of  five 
development  phases 

Cost  (Less  cost 
overrun) 

Large;  Complex;  all 
NASA 

Herbsleb  (1994) 

13  CMM 
Companies 

Process 
Improvement 
ROI  4.0  —  8.8 

CMM  Process 

Areas 

Cost  (Cost 
reduction  through 

SE  investment) 

Various;  federal 
contracting 

Honour  (2004) 
Survey  INCOSE 
SEs 

15-20%  of 
project  should 
be  SE 

Overall  SE  level  of 
effort  (Cost)  & 
related  SE  quality 

Cost  &  Schedule 

Various  sizes 
(measured  by  total 
project  cost) 

Boehm  &  Valerdi 
(2006) 

COCOMO  II 

SE  importance 
grows  with 
project  size 

COCOMO  II  RESL 
(Architecture  and 
Risk) 

Cost 

Various  sizes,  but 
software  systems 
only 

Boehm  &  Valerdi 
(2004) 

COSYSMO 

Estimate 
within  30% 
effort  50%  - 
70%  of  time 

33  activities  defined 
by  EIA  632 

Cost 

Mostly  successful 
projects  from 
federal  contractors 

Ancona  & 

Caldwell  (1990) 

Boundary 

Management 

Managing  team 
boundary  15%; 
more  is  better 

Team  boundary 
activities  —  interface 
between  team  and 
external 

Product 

Performance 
(Successfully 
marketed  products) 

Technology 

products 

Frantz  (1995) 
Boeing  side-by- 
side  projects 

More  SE 
yielded  better 
quality  & 
shorter 
duration 

Defined  by  Frantz 

Product 

Performance  & 
Schedule  (Quality 
of  product  and 
duration  of  project) 

Three  similar 
systems  for 
manipulating 
airframes  during 
assembly 

Mink,  2007 
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NATIONAL  DITENSE  INDUSTRIAL  ASS 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


The  Rigor 

Followed^  Planned^  Lifecycle 


Form  Team 


Develop  Approach 
Validate  Approach 


NDIA 


Survey  (Industry  Projects) 


Pilot  Study  of  Survey 


Collect  Data 


Analyze  Data 


Anonymous  via  Web 
46+  Projects,  by  SEI 


Publish  Results 


Two  Reports: 

1.  Public  Report 

2.  Restricted 
Attachment 


This  study  spanned  three  years 


6 


STRENGTH  THHOIGH  INDUSTRY  &  TECHNOLOGY 


The  Rigor 

Formally  Selected  Set  of  SE  Activities 


CMMI-SW/SE  vi.i 

•  25  Process  Areas 

•  179  Goals 

•  614  Practices 

•  476  Work  Products 


Systems 
Engineering- 
related  Filter 


Size  Constraint 
Filter 


Considered  significant 
to  Systems  Engineering 


•  14  Process  Areas 

•  31  Goals 

•  87  Practices 

•  199  Work  Products 


•  1 3  Process  Areas 

•  23  Goals 

•  45  Practices 

•71  Work  Products 


Survey  was  developed  based  on  standards 
and  recognized  SE  experts 
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STRENGTH  THHOUGH  INDUSTRY  &  TECHNOLOGY 


The  Rigor 

Validated^  Survey  Responses 


Maximum  =  2.8 
3rd  Quartile  =  2.1 
Median  =  1.9 
1st  Quartile  =  1.7 
Minimum  =  1.1 
N  =  64 


Maximum  =  3.9 
3rd  Quartile  =  3.3 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  2.1 
N  =  63 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1.5 
N  =  64 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.75 
1st  Quartile  =  2.3 
Minimum  =  1.7 
N  =46 


Analyzed  distributions,  variability,  relationships... 
To  ensure  statistical  rigor  and  relevance 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Results 

Overall  SE  Capability  &  Project  Performance 


Overall  SE  Capability 


1 .00 


0.75 


0.50 


0.25 


0.00 


Lower  Moderate  Higher 


□  Higher  Project 
Performance 

□  Moderate  Project 
Peformance 

■  Lower  Project 
Performance 


Projects  with  better  Overall  Systems  Engineering 
Capability  delivers  better  Project  Performance 
(cost,  schedule  and  scope) 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

1.  Product  Architecture  and  Project  Performance 


Product  Architecture  Capability 


NATIONAL  DITENSE  INDUSTRIAL  ASS 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

2.  Trade  Studies  and  Project  Performance 


Trade  Studies  Capability 


0 

o 

03 

E 

£ 

0 

□_ 


i.oo- 


0.75- 


0.50- 


0.25- 


0.00- 


17% 

17% 

44% 

41% 

39% 

42% 

49' 

H. 

32% 

19' 

H. 

Lower  Moderate  Higher 

Projects  with  better  Trade  Studies  Capability 
Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

3.  Technical  Solution  and  Project  Performance 


1.00- 


0.75- 

0.50- 

0.25- 

0.00- 


Technical  Solution  Capability 

4 


Lower  Moderate  Higher 


Projects  with  better  Technical  Solution  Capability 
Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Results 

4.  IPTs  and  Project  Performance 


IPT  Capability 


1.00 


0 

o 

03 

E 

£ 

0 

□_ 


13% 


0.75- 


0.50- 


54% 


0.25  - 


0.00  -■ 


33% 


38% 

i 

431 

27% 


20% 


Lower  Moderate  Higher 


Projects  with  better  IPTs  Capability 
Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 


Performance 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

5.  Requirements  and  Project  Performance 


A  100-. 

0.75 

0.50 
0.25  J 
0.00 


Requirements  Capability 


18% 

21% 

55% 

38% 

53% 

44% 

18% 

26% 

27% 

Low 


Moderate 


High 


Projects  with  better  Requirements  Management  and  Development 
Capability  Show  a  “Moderately  Strong  /  Strong”  Positive  Relationship 

with  Performance 


STRENGTH  THHOIGH  INDUSTRY  &  TECHNOLOGY 


Results 

Summary.  OL  Relationships 


SE  Capability' 

Architecture 
Trade  Studies 
Tech  Solution 
IPT 

Requirements 
Risk 
Validation 
Verification 
Prod  Integration 
Proj  Planning 
Config  Mngt 
Monitor  &  Ctrl 


Negative 

40%  -30%  -20%  -10% 


0%  10% 


Positive 

20%  30% 


40%  50% 


Relationship  to  Performance  (Gamma) 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Conclusions  &  Caveats 

Summary 


SE  Effectiveness 

•  Provides  credible  measured  evidence  about  the  value  of 
disciplined  Systems  Engineering 

•  Affects  success  of  systems-development  projects 

Specific  Systems  Engineering  Best  Practices 

•  Highest  relationships  to  activities  on  the  “left  side  of  SE  Vee” 

Environment  (Project  Challenge)  affects  performance  too 

-  Some  projects  are  more  challenging  than  others  ...  and  higher 
challenge  affects  performance  negatively 

-  Yet  good  SE  practices  remain  crucial  for  both  high  and  low 
challenge  projects 
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Conclusions  &  Caveats 

Next  Steps 


•  Correlate  Report  Findings  with  Other  Sources 


•  Develop  Improvement  Recommendations 

•  Policy,  guidance,  training,  measures,  reviews 

•  Conduct  Additional  Analysis  of  Collected  Data 

•  iv  &  v 

•  Discover  other  relationships  and  correlations 

*  Repeat  the  Survey  to  Gauge  Improvements 


Survey  Acquirers 
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Questions? 


Al  Mink 
al_mink@sra.  com 
571  212-4778 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Backup 


NDIA  SE  Effectiveness  Survey 
Analysis  Slides 


Reference:  “A  Survey  of  Systems  Engineering  Effectiveness”,  Software  Engineering  Institute,  Carnegie  Mellon  University, 
CMU/SEI-2007-SR-008.  Joseph  P.  Elm,  Dennis  R.  Goldenson,  Khaled  El  Emam,  Nicole  Donatelli,  Angelica  Nissa. 


Conclusions  &  Caveats 


NATIONAL  DITENSE  INDUSTRIAL  ASSOCIATION 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY  Consistent  with  ‘Top  10  Reasons  Projects  Fail*” 

1.  Lack  of  user  involvement 

2.  Changing  requirements 

3.  Inadequate  Specifications 

4.  Unrealistic  project  estimates 

5.  Poor  project  management 

6.  Management  change  control 

7.  Inexperienced  personnel 

8.  Expectations  not  properly  set 

9.  Subcontractor  failure 

10. Poor  architectural  design 


— 

Above  Items  Can  Cause  Overall 
Program  Cost  and  Schedule  to  Overrun 


*  Project  Management  Institute 


Matching  items  noted  in  RED  22 


Conclusions  &  Caveats 


NATIONAL  DITENSE  INDUSTRIAL  ASSOCIATION 


SHBEJSg-  Consistent  wjth  “Top  5  SE  Issues *”  (2006) 

•  Key  systems  engineering  practices  known  to  be  effective  are  not 
consistently  applied  across  all  phases  of  the  program  life  cycle. 

•  Insufficient  systems  engineering  is  applied  early  in  the  program  life 
cycle,  compromising  the  foundation  for  initial  requirements  and 
architecture  development. 

•  Requirements  are  not  always  well-managed,  including  the  effective 
translation  from  capabilities  statements  into  executable 
requirements  to  achieve  successful  acquisition  programs. 

•  The  quantity  and  quality  of  systems  engineering  expertise  is 
insufficient  to  meet  the  demands  of  the  government  and  the  defense 
industry. 

•  Collaborative  environments,  including  SE  tools,  are  inadequate  to 
effectively  execute  SE  at  the  joint  capability,  system  of  systems,  and 
system  levels. 


*  OUSD  AT&L  Summit 


Matching  items  noted  in  RED 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Summary  SE  Relationships 
to  Project  Performance 


Gamma 


Details 

Project  Challenge 

►J  PC  I  -31  %|  5~0%] 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

|  1.0  |  22%  |  28%  |  50%  |  1.85  | 


Relative  Project  Performance 


Moderate 

Min. 

Range 

Lo 

#  Med 

^ r 

Hi 

Max. 

Range 

|  42%  | 

|  58%  | 

Higher 

Min. 

Range 

Lo 

#  Med 

^ r 

Hi 

Max. 

Range 

r^~i 

00 

00 

|  38%  | 

|  25%  | 

Project  Environment 


CMMI 

22% 

13.0% 

IMP 

5% 

39.0% 

EXP 

9% 

33.0% 

1.0 

36% 

57% 

7% 

1.95 

1.0 

25% 

55% 

20% 

2.17 

1.0 

29% 

42% 

29% 

2.5 

2.7 

33% 

28% 

39% 

4.0 

2.84 

33% 

25% 

42% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

1.95 

29% 

36% 

35% 

2.7 

2.17 

42% 

29% 

29% 

2.84 

2.5 

39% 

44% 

17% 

3.5 

►I 

►I 

►I 

►j 

►I 

id 

id 

id 

id 

id 

id 

id 

id 

id 


Systems  Engineering  Capability 
IPT 
PP 
PMC 
RSKM 
REQ 
TRADE 
ARCH 
TS 
PI 

VER 
VAL 
CM 

Overall  SEC 
REQ+TS 


1.0 

33% 

54% 

13% 

2.5 

1.0 

33% 

54% 

13% 

2.8 

1.0 

23% 

54% 

23% 

2.5 

1.0 

35% 

47% 

18% 

2.8 

1.0 

44% 

38% 

18% 

2.8 

1.0 

39% 

44% 

17% 

2.7 

1.0 

45% 

44% 

11% 

2.7 

1.0 

40% 

53% 

7% 

2.8 

1.0 

36% 

54% 

14% 

1.5 

1.0 

31% 

62% 

7% 

2.7 

1.0 

54% 

23% 

23% 

2.7 

1.0 

29% 

47% 

24% 

3.0 

1.0 

39% 

46% 

15% 

2.5 

1.0 

43% 

50% 

13% 

2.8 

2.5 

43% 

38% 

19% 

3.1 

2.8 

29% 

35% 

36% 

3.3 

2.5 

23% 

46% 

31% 

3.0 

2.8 

27% 

66% 

7% 

3.6 

2.8 

26% 

53% 

21% 

3.4 

2.7 

42% 

41% 

17% 

3.3 

2.7 

29% 

42% 

29% 

3.3 

2.8 

33% 

40% 

27% 

3.2 

1.5 

33% 

38% 

29% 

3.5 

2.7 

33% 

34% 

33% 

3.2 

2.7 

17% 

66% 

17% 

3.3 

3.0 

46% 

36% 

18% 

3.67 

2.5 

29% 

59% 

12% 

3.0 

2.8 

23% 

62% 

15% 

3.1 

3.1 

20% 

27% 

53% 

4.0 

3.3 

35% 

29% 

36% 

4.0 

3.0 

45% 

25% 

30% 

4.0 

3.6 

36% 

0% 

64% 

4.0 

3.4 

27% 

18% 

55% 

4.0 

3.3 

19% 

32% 

49% 

4.0 

3.3 

23% 

31% 

46% 

4.0 

3.2 

27% 

27% 

46% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

3.2 

33% 

20% 

47% 

4.0 

3.3 

29% 

33% 

38% 

4.0 

3.67 

28% 

33% 

39% 

4.0 

3.0 

31% 

13% 

56% 

4.0 

3.1 

22% 

28% 

50% 

4.0 

Acquirer  Capability 

AC  |  -35% |  3~0%1  |  1.0  |  7%|  60% |  33%|  2.5  | 


|  2.5  |  41  %|  32% |  26% |  3.0  | 


|  3.0  |  50% |  25% |  25% |  4.0  | 


Combined  Capability  and  Challenge 

►  |  REQ+TS+PC  |  63%|  0~0%]  |  1.0  |  67%|  33%|  0%|  1.7  | 


|  1.7  |  25% |  45% |  30%|  2.3  | 


|  2.3  |  14%  |  36% |  50% |  4.0  | 


Gamma  relationship  Chance  probability  _ Gamma  relationship  Chance  probability 


Strong 

Very  low 

|  |  Moderately  strong 

Moderately  low 

Moderately  strong 

Low 

|  |  Weak 

Fair 

to  strong 
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Summary  SE  Relationships 
to  Project  Performance 


Relative  Project  Performance 


Gamma 


Details 

Project  Challenge 

►i  pc  r 


Project  Environment 


►I 

►I 

►I 

►I 

id 

id 

id 

id 

id 

■ 

id 

id 

id 

id 


CMMI 

22% 

13.0% 

IMP 

5% 

39.0% 

EXP 

9% 

33.0% 

Systems  Engineering  Capability 

IPT 

34% 

4.0% 

PP 

13% 

25.0% 

PMC 

-13% 

25.0% 

RSKM 

28% 

6.1% 

REQ 

33% 

4.0% 

TRADE 

37% 

3.0% 

ARCH 

40% 

0.2% 

TS 

36% 

3.0% 

PI 

21% 

16.0% 

VER 

25% 

9.0% 

VAL 

28% 

7.0% 

CM 

13% 

26.0% 

Overall  SEC 

32% 

4.0% 

REQ+TS 

49% 

0.5% 

Acquirer  Capability 

C 


AC 


-35 


Combined  Capability  am 
►|  REQ+TS+PC  P 


63 


Lower 

Min. 

^ r 

W 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

Moderate 

Min. 

Range 

Lo 

#  Med 

Hi 

Max. 

Range 

hid 

|  42%  | 

|  58%  | 

Higher 

Min. 

Range 

Lo 

#  Med 

^ r 

Hi 

Max. 

Range 

r^~i 

00 

00 

o 

00 

CO 

|  25%  | 

1.0 

36% 

1.0 

25% 

1.0 

29% 

Highest  scoring  SE  capability  areas  in  Higher  Performing  Projects*: 
Risk  Management;  Requirements  Development  and  Management;  IPTs 

|  *Based  on  small  partitioned  sample  size 


1.0 

33% 

54% 

13% 

2.5 

1.0 

33% 

54% 

13% 

2.8 

1.0 

23% 

54% 

23% 

2.5 

1.0 

35% 

47% 

18% 

2.8 

1.0 

44% 

38% 

18% 

2.8 

1.0 

39% 

44% 

17% 

2.7 

1.0 

45% 

44% 

11% 

2.7 

1.0 

TW 

53% 

7% 

2.8 

1.0 

36% 

54% 

14% 

1.5 

1.0 

31% 

62% 

7% 

2.7 

1.0 

54% 

23% 

23% 

2.7 

1.0 

29% 

47% 

24% 

3.0 

1.0 

39% 

46% 

15% 

2.5 

1.0 

43% 

50% 

13% 

2.8 

2.5 

43% 

38% 

19% 

3.1 

2.8 

29% 

35% 

36% 

3.3 

2.5 

23% 

46% 

31% 

3.0 

2.8 

27% 

66% 

7% 

3.6 

2.8 

26% 

53% 

21% 

3.4 

2.7 

42% 

41% 

17% 

3.3 

2.7 

29% 

42% 

29% 

3.3 

2.8 

33% 

40% 

27% 

3.2 

1.5 

33% 

38% 

29% 

3.5 

2.7 

33% 

34% 

33% 

3.2 

2.7 

17% 

66% 

17% 

3.3 

3.0 

46% 

36% 

18% 

3.67 

2.5 

29% 

59% 

12% 

3.0 

2.8 

23% 

62% 

15% 

3.1 

Lowest  scoring  SE  capability  areas  in  Lower  Performing  Projects*: 
Validation;  Architecture;  Requirements  Development  and  Management 


3.1 

20% 

27% 

53% 

4.0 

3.3 

35% 

29% 

36% 

4.0 

3.0 

45% 

25% 

30% 

4.0 

3.6 

36% 

0% 

64% 

4.0 

3.4 

27% 

18% 

55% 

4.0 

3.3 

19% 

32% 

49% 

4.0 

3.3 

23% 

31% 

46% 

4.0 

3.2 

27% 

27% 

46% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

3.2 

33% 

20% 

47% 

4.0 

3.3 

29% 

33% 

38% 

4.0 

3.67 

28% 

33% 

39% 

4.0 

3.0 

31% 

13% 

56% 

4.0 

3.1 

22% 

28% 

50% 

4.0 

|  25% |  25% |  4.0  | 


|  36% |  50%|  4.0  | 


Gamma  relationship  Chance  probability  _ Gamma  relationship  Chance  probability 


Strong 

Very  low 

|  |  Moderately  strong 

Moderately  low 

Moderately  strong 

Low 

|  |  Weak 

Fair 

to  strong 
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Terminology  and  Notation 

Distribution  Graph 


Histogram  of 
response 
frequencies 


Outliers 


Interquartile 
Range 


Median 


Maximum  =  3.8 
3rd  Quartile  =  3.2 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1.0 
N  =  64 


V. 


“"V- 

Data 

Range 


Sample  size 

(responses  to  corresponding 
survey  questions) 
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Terminology  and  Notation 

Mosaic  Chart 


Column  width 
represents  proportion 
of  projects  with 
this  level  of 


(Lowest,  Intermediate,  Highest) 


Measures  of 


Sample  size  and  distribution  for 
associated  survey  responses 
(capability  +  performance) 


association 
and  statistical  test 


Relative  performance 
distribution  of  the 
sample 


Gamma:  measures  strength  of 
relationship  between  two  ordinal 
variables 

probability  that  an  associative 
relationship  would  be  observed 
by  chance  alone 
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SE  Capability: 

Product  Architecture  (ARCH) 


Maximum  =  4.0 
3rd  Quartile  =  3.5 
Median  =  2.8 
1st  Quartile  =  2.6 
Minimum  =  2.0 
N  =  57 


1.00 


0.75 


0.50 


0.25 


0.00 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(x*2.7)  (2.7<x<  3.3)  (x*3.3) 

N  =  18  N  = 1 4  N  = 1 3 


Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x  £  3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.40 
p  =  0.002 


Relationship  to  project  performance: 


Moderately  strong  to  strong  positive 
relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

45% 

44% 

11% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

23% 

31% 

46% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

29% 

42% 

29% 

3.3 

Gamma 

P 

40% 

0.2% 
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SE  Capability: 

Product  Architecture  (ARCH) 


Survey  Questions 


ID 

Question 

Response  range 

IF01 

This  project  maintains  accurate  and  up-to-date  descriptions  (e.g.  interface  control 
documents,  models,  etc.)  defining  interfaces  in  detail 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF02 

Interface  definition  descriptions  are  maintained  in  a  designated  location,  under 
configuration  management,  and  accessible  to  all  who  need  them 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03a 

For  this  project,  the  product  high-level  structure  is  documented,  kept  up  to  date,  and 
managed  under  configuration  control 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03b 

For  this  project,  the  product  high-level  structure  is  documented  using  multiple  views  (e.g. 
functional  views,  module  views,  etc. 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03c 

For  this  project,  the  product  high-level  structure  is  accessible  to  all  relevant  project 
personnel 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF04 

This  project  has  defined  and  documented  guidelines  for  choosing  COTS  product 
components 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Configuration  Management  (CM) 


Maximum  =  4.0 
3rd  Quartile  =  4.0 
Median  =  3.6 
1st  Quartile  =  3.0 
Minimum  =  2.0 
N  =  58 


Lower 

Capability 

(«3) 

N=  17 


Moderate 
Capability 
(3  <  x  <  4) 
N  =  1 1 


Higher 
Capability 
(x=4) 
N=  18 


Gamma  =  0.1 3 

p  =  0.26 


Relationship  to  project  performance: 


Weak  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

29% 

47% 

24% 

3.0 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.67 

28% 

33% 

39% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.0 

46% 

36% 

18% 

3.67 

Gamma 

P 

13% 

26.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

Configuration  Management  (CM) 


ID 

Question 

Response  Range 

V&V06 

This  project  has  a  configuration  management  system  that  charters  a  Change  Control 

Board  to  disposition  change  requests 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V07 

This  project  maintains  records  of  requested  and  implemented  changes  to  configuration- 
managed  items 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V08 

This  project  creates  and  manages  configuration  baselines  (e.g.,  functional,  allocated, 
product) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

IPT-Related  Capability  (IPT) 


Maximum  =  4.0 
3rd  Quartile  =  3.5 
Median  =  3.0 
1st  Quartile  =  2.5 
Minimum  =  1.0 
N  =  64 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(X£  2.5)  (2.7  <  X  <  3.1)  (xs3.1) 

N=15  N=16  N=15 


Best  Performance 
(x  >  3.0) 


Moderate 
Performance 
(2.5  <  x^3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.34 
p=  0.04 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

33% 

54% 

13% 

2.5 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.1 

20% 

27% 

53% 

4.0 

32 


Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.5 

43% 

38% 

19% 

3.1 

Gamma 

P 

34% 

4.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

IPT-Related  Capability  (IPT) 


ID 

Question 

Response  range 

Proj03 

This  project  uses  integrated  product  teams  (IPTs) 

•Yes 

•No 

Proj04 

This  project  makes  effective  use  of  integrated  product  teams  (IPTs) 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 

Proj06 

My  suppliers  actively  participate  in  IPTs 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 

ProjOYa 

This  project  has  an  IPT  with  assigned  responsibility  for  systems  engineering 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 

ProjOYb 

This  project  has  Systems  Engineering  representation  on  each  IPT 

•highly  compliant 
•largely  compliant; 

•moderately  compliant 
•not  compliant 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Product  Integration  (PI) 


Maximum  =  4.0 
3rd  Quartile  =  3.0 
Median  =  3.0 
1st  Quartile  =  2.0 
Minimum  =  2.0 
N  =  57 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(x  =  1)  (x=  2  or  3)  (x=  4) 

N  =  1  4  N  =  24  N  =  7 


Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x^3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.21 
p=  0.16 


Relationship  to  project  performance: 


Weak  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

36% 

54% 

14% 

1.5 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.5 

29% 

29% 

42% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.5 

33% 

38% 

29% 

3.5 

Gamma 

P 

21% 

16.0% 

STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Survey  Question 


SE  Capability: 

Product  Integration  (PI) 


ID 

Question 

Response  range 

IF05 

This  project  has  accurate  and  up-to-date  documents  defining  its  product  integration 
process,  plans,  criteria,  etc.  throughout  the  life  cycle 

•strongly  disagree  t 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Monitoring  and  Control  (PMC) 


Maximum  =  3.8 
3rd  Quartile  =  3.2 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1.0 
N  =  64 


Lower 
Capability 
(xs  2.5) 
N  =  13 


Moderate 
Capability 
(2.5  <x  <  3) 
N  =  1  3 


Higher 
Capability 
(x*3) 
N=  20 


Gamma  =  -0.1 3 
p=  0.25 


Relationship  to  project  performance:  Weak  negative  relationship 


SE  Capability 


PMC 


Gamma 

P 

-13% 

25.0% 

Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

23% 

54% 

23% 

2.5 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.5 

23% 

46% 

31% 

3.0 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.0 

45% 

25% 

30% 

4.0 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Monitoring  and  Control  (PMC) 


Survey  Questions  (Part  1) 


ID 

Question 

Response  range 

Conti  3 

Do  you  separately  cost  and  track  systems  engineering  activities? 

Yes 

No 

Conti  4a 

Approximately  what  percentage  of  non-recurring  engineering  (NRE)  does  systems 
engineering  represent? 

Percentages  quantized  as: 

•<=  5% 

•<=  10% 

•<=  15% 

•<=  25% 

•>  25% 

Conti  4b 

Is  the  NRE  percentage  estimated,  or  is  it  a  measured  value? 

•estimated 

•measured 

PerfOl 

This  project  creates  and  manages  cost  and  schedule  baselines 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

Perf02b 

EVMS  data  are  available  to  decision  makers  in  a  timely  manner  (i.e.  current  within  2 
weeks) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

Perf02c 

The  requirement  to  track  and  report  EVMS  data  is  levied  upon  the  project’s  suppliers 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

Perf02d 

Variance  thresholds  for  CPI  and  SPI  variance  are  defined,  documented,  and  used  to 
determine  when  corrective  action  is  needed 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Monitoring  and  Control  (PMC) 


Survey  Questions  (Part  2) 


ID 

Question 

Response  range 

Perf02e 

EVMS  is  linked  to  the  technical  effort  through  the  WBS  and  the  IMP/IMS 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

OPerf05 

Does  this  project  track  reports  of  problems  from  fielded  items? 

•Yes 

•No 

Scored  by  • 

the  number 
of  positive 
responses 

OPerf06 

Does  the  project  conduct  an  engineering  assessment  of  all  field  trouble  reports? 

•Yes 

•No 

OPerfOY 

The  results  of  this  engineering  assessment  feed  into  ... 

•operational  hazard 

risk  assessments 

•materiel  readiness 

assessments 

•system  upgrades 

planning 

•other 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Planning  (PP) 


i 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.6 
Minimum  =  2.0 
N  =  63 


1.00- 


0.75- 


0.50- 


0.25- 


0.00-' 


13% 

36% 

36% 

54% 

35% 

29% 

33% 

35% 

29% 

Lower  Moderate  Higher 

Capability  Capability  Capability 

(x^  2.8)  (2.8<  x  <  3.3)  (x ^  3.3) 

N  =  1 5  N  =  1 4  N  =  1 7 


Best 

Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x  <  3.0) 

Lower 

Performance 
(x  <  2.5) 


Gamma  =  0.13 
p  =  0.25 


Relationship  to  project  performance: 


Weak  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

33% 

54% 

13% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

35% 

29% 

36% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

29% 

35% 

36% 

3.3 

Gamma 

P 

13% 

25.0% 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  1) 


SE  Capability: 

Project  Planning  (PP) 


ID 

Question 

Response  range 

PD01 

This  project  utilizes  a  documented  set  of  systems  engineering  processes  for  the  planning 
and  execution  of  the  project 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02a 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
includes  task  descriptions  and  work  package  descriptions 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02b 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that  is 
based  upon  the  product  structure 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02c 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that  is 
developed  with  the  active  participation  of  those  who  perform  the  systems  engineering 
activities 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD02d 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that  is 
developed  with  the  active  participation  of  all  relevant  stakeholders,  e.g.,  developers, 
maintainers,  testers,  inspectors,  etc. 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD03a 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and  methodology  to  create 
the  initial  conceptual  design  for  product  development)  is  complete,  accurate  and  up-to- 
date 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD03b 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and  methodology  to  create 
the  initial  conceptual  design  for  product  development)  is  developed  with  the  active 
participation  of  those  who  perform  the  systems  engineering  activities 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD03c 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and  methodology  to  create 
the  initial  conceptual  design  for  product  development)  is  developed  with  the  active 
participation  of  all  appropriate  functional  stakeholder 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Planning  (PP) 


Survey  Questions  (Part  2) 


ID 

Question 

Response  range 

PD04a 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is  an 
event-driven  plan  (i.e. ,  each  accomplishment  is  tied  to  a  key  project  event) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD04b 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that 
documents  significant  accomplishments  with  pass/fail  criteria  for  both  business  and 
technical  elements  of  the  project 

•strongly  disagree  j 

•disagree 

•agree 

•strongly  agree 

PD04c 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is 
consistent  with  the  WBS 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05a 

This  project  has  an  integrated  event-based  schedule  that  is  structured  as  a  networked, 
multi-layered  schedule  of  project  tasks  required  to  complete  the  work  effort 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05b 

This  project  has  an  integrated  event-based  schedule  that  contains  a  compilation  of  key 
technical  accomplishments  (e.g.,  a  Systems  Engineering  Master  Schedule) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05c 

This  project  has  an  integrated  event-based  schedule  that  references  measurable  criteria 
(usually  contained  in  the  Integrated  Master  Plan)  required  for  successful  completion  of 
key  technical  accomplishments 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD05d 

This  project  has  an  integrated  event-based  schedule  that  is  consistent  with  the  WBS 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Project  Planning  (PP) 


Survey  Questions  (Part  3) 


ID 

Question 

Response  range 

PD05e 

This  project  has  an  integrated  event-based  schedule  that  identifies  the  critical  path  of  the 
program  schedule 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD06 

This  project  has  a  plan  or  plans  for  the  performance  of  technical  reviews  with  defined 
entry  and  exit  criteria  throughout  the  life  cycle  of  the  project 

•strongly  disagree  j 

•disagree 

•agree 

•strongly  agree 

PD07 

This  project  has  a  plan  or  plans  that  include  details  of  the  management  of  the  integrated 
technical  effort  across  the  project  (e.g.,  a  Systems  Engineering  Management  Plan  or  a 
Systems  Engineering  Plan) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD08 

Those  who  perform  systems  engineering  activities  actively  participate  in  the  development 
and  updates  of  the  project  planning 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD09 

Those  who  perform  systems  engineering  activities  actively  participate  in 
tracking/reporting  of  task  progress 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

42 


STRENGTH  THHOLGH  INDUSTRY  &  TCCHNOLOGY 


SE  Capability: 

Requirements  Development  &  Mgmt  (REQ) 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.8 
Minimum  =  2.2 
N  =  58 


1.00_ 

0.75 

0.50_ 
0. 25  _ 
0.00 


18% 

21% 

55% 

38% 

53% 

44% 

18% 

26% 

27% 

Lower 

Moderate 

High 

Capability 

Capability 

Capat 

(x  <  2.S) 

(2.0  <x  <  3.4) 

(x  >: 

N=  16 

N=  19 

N  =  ■ 

Best  Performance 
(x  >3.0) 

Moderate 
Performance 
(25  <  x  <3.01 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.33 
p=  0.04 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

44% 

38% 

18% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.4 

27% 

18% 

55% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

26% 

53% 

21% 

3.4 

Gamma 

P 

33% 

4.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Requirements  Development  &  Mgmt  (REQ) 


Survey  Questions  (Part  1) 


ID 

Question 

Response  range 

RDOIa 

This  project  maintains  an  up-to-date  and  accurate  listing  of  all  requirements  specified  by 
the  customer,  to  include  regulatory,  statutory,  and  certification  requirements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RDOlb 

This  project  maintains  an  up-to-date  and  accurate  listing  of  all  requirements  derived  from 
those  specified  by  the  customer 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD02 

This  project  maintains  up-to-date  and  accurate  documentation  clearly  reflecting  the 
hierarchical  allocation  of  both  customer  and  derived  requirements  to  each  element 
(subsystem,  component,  etc.)  of  the  system  in  the  configuration  baselines 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD03a 

This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of 
operational  concepts  and  their  associated  scenarios 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD03b 

This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  use  cases 
(or  their  equivalent) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD03c 

This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  product 
installation,  maintenance  and  support  concepts 

•strongly  disagree  \ 

•disagree 

•agree 

•strongly  agree 

RD04 

This  project  has  documented  criteria  for  identifying  authorized  requirements  providers  to 
avoid  requirements  creep  and  volatility 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Requirements  Development  &  Mgmt  (REQ) 


Survey  Questions  (Part  2) 


ID 

Question 

Response  range 

RD05 

This  project  has  documented  criteria  (e.g.,  cost  impact,  schedule  impact,  authorization  of 
source,  contract  scope,  requirement  quality)  for  evaluation  and  acceptance  of 
requirements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD06 

The  requirements  for  this  project  are  approved  in  a  formal  and  documented  manner  by 
relevant  stakeholders 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD07 

This  project  performs  and  documents  requirements  impact  assessments  for  proposed 
requirements  changes 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD08 

This  project  develops  and  documents  project  requirements  based  upon  stakeholder 
needs,  expectations,  and  constraints 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD09 

This  project  has  an  accurate  and  up-to-date  requirements  tracking  system 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RDlOa 

For  this  project,  the  requirements  documents  are  managed  under  a  configuration  control 
process 

•strongly  disagree  \ 

•disagree 

•agree 

•strongly  agree 

RDlOb 

For  this  project,  the  requirements  documents  are  accessible  to  all  relevant  project  staff 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Risk  Management  (RSKM) 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.8 
Minimum  =  2.2 
N  =  58 


Lower 
Capability 
(xs  2.8) 
N=  17 


0% 

1/ 


Moderate  Higher 

Capability  Capability 

(2.7  <  K  <  3.6)  (xs3.6) 

N=15  N  =  1  4 


Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x£3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.28 

p  =  0.061 


Relationship  to  project  performance:  Moderately  strong  positive  relationship 


SE  Capability 


RSKM 


Gamma 

P 

28% 

6.1% 

Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

35% 

47% 

18% 

2.8 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

27% 

66% 

7% 

3.6 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.6 

36% 

0% 

64% 

4.0 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

Risk  Management  (RSKM) 


ID 

Question 

Response  range 

PDIIa 

This  project  has  a  Risk  Management  process  that  creates  and  maintains  an  accurate  and 
up-to-date  list  of  risks  affecting  the  project  (e.g.,  risks  to  cost,  risks  to  schedule,  risks  to 
performance) 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PDIIb 

This  project  has  a  Risk  Management  process  that  creates  and  maintains  up-to-date 
documentation  of  risk  mitigation  plans  and  contingency  plans  for  selected  risks 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PDIIc 

This  project  has  a  Risk  Management  process  that  monitors  and  reports  the  status  of  risk 
mitigation  activities  and  resources 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PDIId 

This  project  has  a  Risk  Management  process  that  assesses  risk  against  achievement  of 
an  event-based  schedule 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

PD12 

This  project's  Risk  Management  process  is  integrated  with  program  decision-making 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Trade  Studies  (TRADE) 


Maximum  =  4.0 
3rd  Quartile  =  3.7 
Median  =  3.0 
1st  Quartile  =  2.3 
Minimum  =  1.0 
N  =  58 


1.00 


0.75 


0.50 


0.25 


0.00 


Best  Performance 
(x  >  3.0) 


Moderate 
Performance 
(2.5  <  x£  3.0) 

Lower  Performance 
(x  <  2.5) 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(x^  2.7)  (2.7  <  x  <  3.3)  (xa3.3) 

N  =  1  8  N  =  1  2  N=16 


Gamma  =  0.37 
p  =  0.03 


Relationship  to  project  performance: 


Moderately  strong  to  strong  positive 
relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

39% 

44% 

17% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

19% 

32% 

49% 

4.0 
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Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

42% 

41% 

17% 

3.3 

Gamma 

P 

37% 

3.0% 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions 


SE  Capability: 

Trade  Studies  (TRADE) 


ID 

Question 

Response  range 

RD11 

Stakeholders  impacted  by  trade  studies  are  involved  in  the  development  and 
performance  of  those  trade  studies 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD12 

This  project  performs  and  documents  trade  studies  between  alternate  solutions  based 
upon  definitive  and  documented  selection  criteria 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD13 

Documentation  of  trade  studies  is  maintained  in  a  defined  repository  and  is  accessible  to 
all  relevant  project  staff 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHtKGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Technical  Solution  (TS) 


Maximum  =  4.0 
3rd  Quartile  =  3.3 
Median  =  2.9 
1st  Quartile  =  2.6 
Minimum  =  2.1 
N  =  57 


Note:  TS  is  a  composite  measure  equivalent  to  ARCH  +  TRADE. 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(xs2.8)  (2.8  <  x  <  3.2)  (xa3.2) 

N=15  N  =  1  5  N  =  1  5 


Gamma  =  0.36 
p  =  0.03 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

40% 

53% 

7% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.2 

27% 

27% 

46% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

33% 

40% 

27% 

3.2 

Gamma 

P 

36% 

3.0% 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  1) 


SE  Capability: 

Technical  Solution  (TS) 


ID 

Question 

Response  Range 

RD11 

Stakeholders  impacted  by  trade  studies  are  involved  in  the  development  and 
performance  of  those  trade  studies 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

RD12 

This  project  performs  and  documents  trade  studies  between  alternate  solutions  based 
upon  definitive  and  documented  selection  criteria 

•strongly  disagree  ' 

•disagree 

•agree 

•strongly  agree 

RD13 

Documentation  of  trade  studies  is  maintained  in  a  defined  repository  and  is  accessible  to 
all  relevant  project  staff 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF01 

This  project  maintains  accurate  and  up-to-date  descriptions  (e.g.  interface  control 
documents,  models,  etc.)  defining  interfaces  in  detail 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF02 

Interface  definition  descriptions  are  maintained  in  a  designated  location,  under 
configuration  management,  and  accessible  to  all  who  need  them 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

51 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  2) 


SE  Capability: 

Technical  Solution  (TS) 


ID 

Question 

Response  Range 

IF03a 

For  this  project,  the  product  high-level  structure  is  documented,  kept  up  to  date,  and 
managed  under  configuration  control 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF03b 

For  this  project,  the  product  high-level  structure  is  documented  using  multiple  views  (e.g. 
functional  views,  module  views,  etc.) 

•strongly  disagree  ' 

•disagree 

•agree 

•strongly  agree 

IF03c 

For  this  project,  the  product  high-level  structure  is  accessible  to  all  relevant  project 
personnel 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

IF04 

This  project  has  defined  and  documented  guidelines  for  choosing  COTS  product 
components 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Validation  (VAL) 


Maximum  =  4.0 
3rd  Quartile  =  3.7 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  1.7 
N  =  58 


Lower  Moderate  Higher 

Capability  Capability  Capability 

(xs2.7)  (2.7  <  x  <  3.3)  (x ^  3.3) 

N  =  1  3  N  =  1  2  N  =  21 


Gamma  =  0.28 
p  =  0.07 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

54% 

23% 

23% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.3 

29% 

33% 

38% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

17% 

66% 

17% 

3.3 

Gamma 

P 

28% 

7.0% 

VAL 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Validation  (VAL) 


Survey  Questions 


ID 

Question 

Response  Rate 

V&V04a 

This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
validation  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V04b 

This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used 
for  the  validation  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V05 

This  project  maintains  a  listing  of  items  managed  under  configuration  control 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Verification  (VER) 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.6 
Minimum  =  2.2 
N  =  58 


7% 


Lower 
Capability 
0^2.7) 
N  =  16 


Moderate  Higher 

Capability  Capability 

(2.7  <  x  <  3.2)  (x^  3.2) 

N=15  N=15 


Gamma  =  0.25 
p=  0.09 


Relationship  to  project  performance: 


Moderately  strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

31% 

62% 

7% 

2.7 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.2 

33% 

20% 

47% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.7 

33% 

34% 

33% 

3.2 

Gamma 

P 

25% 

9.0% 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Verification  (VER) 


Survey  Questions  (Part  1) 


ID 

Question 

Response  range 

V&VOIa 

This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
test  and  verification  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&VOlb 

This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used  for 
the  verification  of  systems  and  system  elements 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02a 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  defines  entry  and  exit  criteria  for  work  products 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02b 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  includes  training  requirements  for  the  reviewers 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02e 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  addresses  identified  risks  and  risk  mitigation  activities  during  reviews 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02f 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  examines  completeness  of  configuration  baselines 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Survey  Questions  (Part  2) 


SE  Capability: 

Verification  (VER) 


ID 

Question 

Response  range 

V&V03 

This  project  conducts  non-advocate  reviews  (e.g.  reviews  by  qualified  personnel  with  no 
connection  to  or  stake  in  the  project)  and  documents  results,  issues,  action  items,  risks, 
and  risk  mitigations 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02C 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  defines  criteria  for  the  selection  of  work  products  (e.g.,  requirements 
documents,  test  plans,  system  design  documents,  etc.)  for  review 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 

V&V02d 

This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  tracks  action  items  to  closure 

•strongly  disagree 

•disagree 

•agree 

•strongly  agree 
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STRENGTH  TUHOLGH  INDUSTRY  &  TCCHNOLOGY 


SE  Capability: 

Combined  Reqts+Tech  Solution  (REQ+TS) 


(This  is  a  higher  order  measure; 
see  base  measures  for  distribution) 


Lower  Moderate  Best 

Capability  Capability  Capability 

(xs2.8)  (2.8  <  x  <  3.1)  (x^  3.1) 

N  =  1  5  N  =  1  3  N  =  1  8 


Gamma  =  0.49 
p  =  0.005 


Relationship  to  project  performance: 


Strong  positive  relationship 


SE  Capability 


Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

43% 

50% 

13% 

2.8 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.1 

22% 

28% 

50% 

4.0 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.8 

23% 

62% 

15% 

3.1 

Gamma 

P 

49% 

0.5% 

REQ+TS 
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NATIONAL  DITENSE  INDUSTRIAL  ASSOCIATION 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Total  Systems  Engineering  Capability 


Maximum  =  3.9 
3rd  Quartile  =  3.3 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  2.1 
N  =  63 


1.00- 


0.75- 


0.50- 


0.25  - 


0.00- 


46% 


Lower 
Capability 
(x  £2.5) 
N  =  13 


12% 

ii - 

59% 

56% 

13% 

29% 

31% 

Moderate 
Capability 
(2.5<x<3.0) 
N  =  17 

Higher 
Capability 
(x  *3.0) 

N  =  16 

Best  Performance 
(x  >  3.0) 

Moderate 
Performance 
(2.5  <  x£3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  0.32 
p  =  0.04 


Relationship  to  project  performance:  Moderately  strong  positive  relationship 


SE  Capability 


Overall  SEC 


Gamma 

P 

32% 

4.0% 

Lower 

Min. 

^ r 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

39% 

46% 

15% 

2.5 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.5 

29% 

59% 

12% 

3.0 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

3.0 

31% 

13% 

56% 

4.0 
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STRENGTH  THHOLGH  INDUSTRY  &  TECHNOLOGY 


Project  Challenge  (PC) 


■ 

Project  challenge  factors: 

•Life  cycle  phases 

•Project  characteristics 

(e.g.,  size,  effort,  duration,  volatility) 

•Technical  complexity 

•Teaming  relationships 


Lower  Moderate  Higher 

Challenge  Challenge  Challenge 

(x  £1 .85)  (1 ,85<x<2.05)  (x  ^2.05)  Gamma  =  -0.31 

N  =  18  N  =  12  N=  16  p=  0.05 


Relationship  to  project  performance: 

Moderately  strong  negative 

relationship 

Project  Challenge 


Gamma 

P 

-31% 

5.0% 

Lower 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

22% 

28% 

50% 

1.85 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.85 

42% 

58% 

0% 

2.05 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.05 

38% 

38% 

25% 

4.0 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Capability: 

Reqts+Tech  Solution  with  Project  Challenge 


1.00H 


0.75 


0.5O 


0.25  H 


0.00  “I 


Lower  Challenge 

CPC  <  1 .95 


Lower 
REQ  +TS 
Capability 
(x  £  2 .3) 

N  =8 

Gamma  =  0.57 
p- value  =  0.02 


Moderate 
REQ+TS 
Capability 
C2.8<x<3.1) 
H  =  6 


Higher  Challenge 

CPC  >1.9) 


Higher 
REQ  +TS 
Capability 
(x  £  3. 1 ) 
H  =7 


Lower 
REQ+TS 
Capability 
(x  ^2.8) 
H  =7 


Moderate 
REQ  +TS 
Capability 
(2.3<x<3.1) 
N  =  7 


Higher 
REQ  +TS 
Capability 
(x  £  3.1) 
H  =11 


Gamma  =0.54 
p-value  =0.03 


Best 

Performance 
(x>  3.0) 

Moderate 
Performance 
(2 .5  <  x  <  3 .0) 

Lower 

Performance 
(x  <  2 .5) 


Relationship  to  project  performance:  Very  strong  positive  relationship 


SE  Capability  +  Project  Challenge 


REQ+TS+PC 


Gamma 

P 

63% 

0.0% 

Lower 

Min. 

^ r 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.0 

67% 

33% 

0% 

1.7 

Moderate 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

1.7 

25% 

45% 

30% 

2.3 

Higher 

Min. 

Max. 

Range 

Lo 

#  Med 

Hi 

Range 

2.3 

14% 

36% 

50% 

4.0 
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STRENGTH  THHtKGH  INDUSTRY  &  TECHNOLOGY 


n 


Relating  Project  Performance  to 
Project  Challenge  and  SE  Capability 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Effectiveness 

Relationship  of  SEC  to  Performance 


Supplier  Systems  Engineering  Capability^ 

Relationship  to 

Project  Performance 

Relationship 

(Gamma^l) 

Section 

Reference 

Project  Planning 

Weak  positive  relationship 

+0.13 

5.1. 3.2 

Project  Monitoring  and  Control 

Weak  negative  relationship 

-0.13 

5.1.3.3 

Risk  Management 

Moderately  strong  positive 
relationship 

+0.28 

5.1. 3.4 

Requirements  Development  &  Management 

Moderately  strong  positive 
relationship 

+0.33 

5.1.3.5 

Trade  Studies 

Strong  positive  relationship 

+0.37 

5.1. 3.6 

Product  Architecture 

Moderately  strong  to  strong  positive 
relationship 

+0.40 

5.1.3.7 

Technical  Solution 

Moderately  strong  positive 
relationship 

+0.36 

5.1.3.8 

Product  Integration 

Weak  positive  relationship 

+0.21 

5.1. 3.9 

Verification 

Moderately  strong  positive 
relationship 

+0.25 

5.1.3.10 

Validation 

Moderately  strong  positive 
relationship 

+0.28 

5.1.3.11 

Configuration  Management 

Weak  positive  correlation 

+0.13 

5.1.3.12 

IPT-Related  Capability 

Moderately  strong  positive  correlation 

+0.34 

5.1.3. 1 
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STRENGTH  THROUGH  INDUSTRY  &  TCCHNOLOGY 


SE  Effectiveness 

Methodology  (In  Detail) 


SEEC  Activities 


NDIASED 
active  roster 


o 


NDIAmg’t  f 
input 


Identify 

industry 

members’ 

focals 


Company  Focal 

Activities 


Respondent 

Activities 


SEI  Activities 


Contact 

Provide 

Focal 

Focal 

focals,  brief 

Web 

contact  #1 

contact  #2 

the  survey 

-> 

access 

-► 

to 

-> 

to 

process, 

data  to 

expedite 

expedite 

solicit  support 

focals 

response 

response 

v _ y 

v _ y 

V _ _ J 

V _ . 

Report* 
findings  to 
NDIA  and 
OSD 


Identify 
respondents 
and  report 
number  to 
SEI 


Solicit 

respondents 
and  provide 
Web  site 
access  info 


Responde 
nt  contact 
#1  to 
expedite 
response 


Responde 
nt  contact 
#2  to 
expedite 
response 


Report 
number  of 
responses 
provided  to 
SEI 


Complete  questionnaire 
and  submit  to  SEI 


Collect  responses  and 
response  rate  data 


Report 
completion 
to  focal 


Analyze  data  and 
report  to  SEEC 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


SE  Effectiveness 

Methodology  (Mathematical  Statement) 


Perf=  f(PC,  PE,  SEC,  AC) 


Perf-  Project  Performance 
PC  -  Project  Challenge 
PE  -  Project  Environment  PE 
SEC  -  Systems  Engineering  Capability 
AC  -  Acquirer  Capability 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Results 

Summary  of  Relationships 


Driving  Factor 

Relationship  to  Project 
Performance 

Description 

r 

Requirements  and 
Technical 

Solution  Combined 
with  Project  Challenge 

Very  strong  positive 

+0.63 

Combined 

Requirements  and 
Technical  Solution 

Strong  positive 

+0.49 

Product  Architecture 

Moderately  strong 
to  strong  positive 

+0.40 

Trade  Studies 

Moderately  strong 
to  strong  positive 

+0.37 

IPT-Related  Capability 

Moderately  strong 
positive 

+0.34 

Technical  Solution 

Moderately  strong 
positive 

+0.36 

Requirements 
Development 
and  Management 

Moderately  strong 
positive 

+0.33 

Driving  Factor 

Relationship  to  Project 
Performance 

Description 

r 

Total  Systems 
Engineering  Capability 

Moderately  strong 
positive 

+0.32 

Project  Challenge 

Moderately  strong 
negative 

-0.31 

Validation 

Moderately  strong 
positive 

+0.28 

Risk  Management 

Moderately  strong 
positive 

+0.28 

Verification 

Moderately  strong 
positive 

+0.25 

Product  Integration 

Weak  positive 

+0.21 

Project  Planning 

Weak  positive 

+0.13 

Configuration 

Management 

Weak  positive 

+0.13 

Process  Improvement 

Weak  positive 

+0.05 

Project  Monitoring  and 
Control 

Weak  negative 

-0.13 
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The  views  expressed  in  this  briefing  are  those  of  the  author  and  do  not  necessarily  reflect 
the  official  policy  or  position  of  the  Air  Force,  The  Department  of  Defense  or  the  U.S.  Government. 


Overview 


•  Background 

•  Systems  Engineering  Challenges 

•  Key  Enablers 

•  The  Way  Ahead 


Background 


•  Five  Years  Ago  Revitalization  of  Systems 
Engineering  Established 

•  Current  Feedback  Indicates  this  has  “...not 
eliminated  cost  and  schedule  problems  for  major 
weapons  development  programs.”* 

•  Shortages  in  Qualified  Systems  Engineers 

•  We  are  NOT  meeting  the  Challenges  of  Current 
Systems  or  Projected  Systems  of  the  Future 


*  GAO  Report  “Major  Weapon  Systems  Continue  to  Experience  Cost  and 
Schedule  Problems  Under  DoD’s  Revised  Policy”,  April  2006. 


Systems  Engineering  Challenges 


•  Inconsistent  Application  of  SE 

•  Need  More  Qualified  SE 

•  Product  Complexity/Product  Control 

•  Poor  Requirements  Up  Front 

•  Where  Does  Sustainment  Fit  In? 

•  Loss  of  Historical  Information 

•  Systems  Engineering  Research 


Inconsistent  Application  of  SE 


Multiple  Definitions  of  SE  Exist 

Harmonization  of  SE  Stds  is  Not  Occurring  “Fast”  Enough 
SE  Stds’  Architecture  needed  to  see  Big  Picture 


-  Consistent  Definition  at  Beginner/Intermediate/Expert  Levels 

-  Simple  to  Complex  Program  Guidance 


-  Domain  Guidance 


Agreement  on  SE  Processes  Definitions  Needed  by 

-  Industry  Associations  (INCOSE,  NDIA,  IEEE,  GEIA) 

-  Standard  Committees  (ISO,  EIA,  IEEE) 

-  Other  Stakeholders  (OSD,  Academia,  SEI) 

Update  and  Implementation  of  SE  Continual  Improvement 
Methodologies  to  Improve  Consistency 


Standardized  Definition  of  SE  Processes  is  Key!! 


Need  More  Qualified  SE 


•  Current  Activities  Ineffective  in  Impacting  Issues* 


•  Need  Innovative  Approach 

-  Identify  Potential  Early  and  Encourage  Development 

-  Introduce  Systems  Thinking  Early** 

1.  Thinking  Broadly,  Big  Picture  5.  Thinking  Outside  the  Box 

2.  Curiosity,  Questioning  6.  Strong  Communication  Skills 

3.  Open-mindedness  7.  Tolerance  for  Uncertainty 

4.  Strong  Interpersonal  Skills  8.  Multi-taskers 

-  Create  Choices  In  Undergraduate  Programs 

•  Emphasize  SE  Development  in  Workforce 

-  Identify  SE  Core  Competencies  and  Performance  Accountability 

-  Create  Programs  to  Develop  High  Potential  Employees 

•  Potential  Graduate  Degrees 

•  Target  Life  and  Work  Experiences  Through  OJT 

Innovative  Academic  Roadmap  Needed  Now 

•  NDIA  “Top  Five  SE  Issues  within  DoD  and  Defense  Industry”,  July  2006 

**  Davidz,  “Enabling  Systems  Thinking  to  Accelerate  the  Development  of  Senior  SE”,  2006 


Product  Complexity/Product  Control 


•  Complexity  of  the  Product  is  Inverse  to  Ease  of  Control 

—  As  Systems  Become  More  Complex  their  Development  is  Harder 
to  Control 

•  Need  to  Identify  SE  Tools  to  Simplify  the  Control  Aspects 
of  Future  Programs 

•  Technical  Management  Processes  must  keep  Pace  with 
Increasing  Complexity 


Collaborative  Environments  must  be 
Integrated  Through-out  Systems  Life 


Poor  Requirements  Up  Front 


•  Not  a  New  Problem* 


•  Need  Cause  and  Effect  Analysis  to  Identify  all  Pitfalls 

•  Need  Design  Solution  Accepted  by  All  with  Recommended 
Changes  to  Existing  Policies  and  Procedures 

•  Need  Methods  to  Highlight  Risk  and  Complexity 

•  No  Connection  Between  Identifiers/Developers  of  Capability 
AND  the  Cost/Schedule  Estimators  UP  FRONT 

Bridges  of  Accountability 
Between  Technical  and  Business 


*  NDIA  “Top  Five  SE  Issues  within  DoD  and  Defense  Industry”,  2004  and  2006. 


Where  Does  Sustainment  Fit  In? 


•  Build  Now  -  Fix  Later  Never  a  Planned  Strategy 

-  Not  Efficient  Enough  for  the  Future 

-  Future  Reduction  of  Personnel  and 

-  Lower  Operating/Sustaining  Budgets  Expected 


•  Information  on  SE  Sustainment  Application  Needed 

-  Guidance  for  SE  Sustainment  Monitor  Activities 

-  Guidance  for  SE  Sustainment  Modification  Activities 

-  Guidance  for  SE  Concept/Technology/Development  Activities 
Focused  on  Sustainment 


•  Change  Title  and  Requirement  for  IOC  to  Initial 
Operational/Support  Capability  (IOSC) 

-  Elevate  to  Milestone  D 

Special  Emphasis  on  Sustainment  from  Start 
Resurgence  of  Acquisition  Logistics 


Loss  of  Historical  Information 


New  Information  Management  Process  in  ISO  1528 
-  Information  stored  in  System  Electronic  Database 


•  Adapt  Mindset  of  Reuse  to  Other  SE  Issues 

-  Decision  Analysis  &  Results  Data 

-  Decisions  and  Paths  Followed 

-  Methods  Used  Lessons  Learned 

-  Time  Tables  of  Changes  With  Initiator  Identified 

•  Program  Linkage  Maintained  Throughout  Lifecycle 


System  Electronic  Database  Transfers  With 
Program/Engineering  Responsibility  through  System  Life 


*  DoD  5000.1  Enclosure  1  Additional  Policy  El. 18  “Products,  Services  and  Technologies”,  last  updated  Feb  2007. 


Systems  Engineering  Research 


SE  Research*  will  Help  get  us  there 
We  Need  Champions! 


•  Complexity/Risk  Impacts  Pre  Program 

•  Expansion  of  SE  Indicators,  Incorporation  of 
Risk/Complexity  into  EVMS 

•  Early  Incorporation  of  Technology/Planning 

•  SE  Friendly  Contracting  Solutions 

•  Evolution  of  SE  Process  Areas 

•  M&S  Tools  and  Management  Environments 
(Toolsets) 

•  Domain  Specific  Complexity  Studies 
(e.g.  SW  Measures  Other  than  SLOC) 


*  Castellano  Briefing,  “Emerging  Research  Ideas  Stemming  from  Systemic  Analysis  of  DoD 
Programs”,  June  2007  And  Masterson  Briefing,  “Systems  Engineering  Research  Needs”,  July  2007 


SE  Challenges  &  Recommendations 

 Summary 


Inconsistent  Application  of  SE 

Agreed  Standardized 

SE  Definition 

Need  More  Qualified  SE 

Innovative  Academic 

Approach 

Product  Complexity/Product  Control 

Collaborative  Environment 

Poor  Requirements  Up  Front 

Bridges  of  Accountability 

Where  Does  Sustainment  Fit? 

Focus  on  Sustainment  & 
Acquisition  Logistics 

Loss  of  Historical  Data 

Electronic  System  Database 

Systems  Engineering  Research 

Champions  Needed 

Key  Enablers 


•  Systems  Thinking  for  all  Functionals  to: 


-  Learn  Technical  Basics  of  System,  Participate  in  Risk  Assessments 
and  Bring  Their  Strategies  to  Table  to  Develop  Overall  Program 
Acquisition  Strategy 

•  Institute  an  Systems  Electronic  Database  for  Life 

-  Decision  Analysis  and  Results;  Risk  Assessment  and  Measures, 

-  Functional  Strategies,  SAMP,  ASP,  RFP,  SSP 

-  All  Functionals  Identify  and  Share  Changes  to  Program  Baselines 

•  Discipline,  Discipline,  Discipline... 


Systems  Engineering  is  NOT  Just  for  Engineers!!! 


The  Way  Ahead 


1.  Complexity  will  Increase  in  Future  Systems 


2.  An  Integrated  and  Coordinated  Effort  is  Needed  Now 

3.  First  Step  is  to  Get  SE  Processes  Defined/Accepted 

4.  Second  Step  is  to  Hold  a  SE  Vision  Forum/Workshop 

-  Attended  by  All  SE  Associated  Organizations 

-  Purpose  to  Define  the  Way 

-  Generate  and  Execute  A  Realization  Plan  for  Global 
Systems  Engineering  for  the  Future 


Volunteer  to  Participate  so  You  Can  be  Heard! 


DRAFT 


DRAFT 


Establish  M&S-related 
Guidelines  for  Solicitations, 
Source  Selections,  and 

Contracting 


Acquisition  Business  Plan 

Action  1-5 


DRAFT 


DRAFT 


Tasking 

Establish  M&S-related  guidelines  for  solicitations, 
source  selections,  and  contracting. 

RATIONALE:  There  are  insufficient  guidelines  regarding  contracting  for  M&S  and  the  data  it  needs  or 
produces.  Acquisition  programs  often  leave  M&S  planning,  use,  and  ownership  to  prime  contractors. 
Government  organizations  are  often  unaware  of  contractor  attributes  that  are  indicators  of  M&S 
capability  maturity  and  are,  therefore,  useful  criteria  in  evaluating  proposals.  Rarely  is  early 
consideration  and  contractual  direction  specifically  intended  to  provide  access  to,  or  reuse  of,  models 
and  data  across  the  life-cycle. 

DISCUSSION:  The  recommended  RFP  language  and  contract  provisions  should  address  M&S 
strategy;  representation  requirements;  M&S  tool  sources;  ownership  and  maintenance;  data  sources 
and  rights;  VV&A;  user  support;  access  control;  and  metrics  and  documentation  requirements,  all 
across  the  system  life-cycle.  The  source  selection  criteria  guidance  should  address  those  contractor 
attributes  that  have  a  direct  relationship  to  successful  M&S  use. 

LEAD:  USD(AT&L)/DS 

SUPPORT:  USD(AT&L)/  Defense  Procurement  and  Acquisition  Policy  (DPAP),  DOT&E  and 
Components 

PRODUCTS:  Sample  language  and  suggested  criteria  in  DAG.  Updates  to  Federal  Acquisition 
Regulations  (FAR)  as  appropriate.  Contract  Data  Requirements  List  (CDRL)  or  a  family  of  CDRLs 
listing  the  M&S  requirements  for  an  RFP. 
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Decomposing  the  Problem 


DRAFT 


“The  PM  shall  plan  for 
M&S  throughout  the 
acquisition  life  cycle.  The 
PM  shall  identify  and  fund 
required  M&S  resources 
early  in  the  life  cycle”  - 
DoDI  5000.2  E5.1.10 


Clear, 

Understandable 

Terms 


How  much  am  I 
willing  to  pay? 


Market  Value 
Weight  of  Requirement 
Qualifications  of  Provider 


Cost  Value 
Value  of  Future  Business 
Risk 


What  am  I  willing 
to  sell  it  for? 


Goods  and 

services 

needed 


Modeling  and  Simulation  Capability 


Modeling  and  Simulation 
Software  /  IT 


Input/Output 
Data  Sets  and 
Descriptions 


Requirements 

Ownership 

Maintenance 


Program  Requirements 
Statutes,  Regulations,  Policies 

Documentation 

Terms  of 
Agreement 


Good  Citizenship 

Specific  Qualities; 
Adherence/Compliance 
With  Standards 


“The  SOW  should  specify 
in  clear,  understandable 
terms  the  work  to  be  done 
in  developing  or  producing 
the  goods  to  be  delivered 
or  services  to  be  performed 
by  a  contractor. 

Preparation  of  an  effective 
SOW  requires  both  an 
understanding  of  the  goods 
or  services  that  are  needed 
to  satisfy  a  particular 
requirement  and  an  ability 
to  define  what  is  required  in 
specific,  performance- 
based,  quantitative  terms.” 
-  MIL-HDBK-245D 


Action  Deliverables 

Objective:  Help  the  Program  Manager  plan  for,  request  and  get  what  they 
need 

-  Raise  questions  for  consideration 

-  Advise  on  appropriateness  of  request  and  completeness  and  quality  of  response 

-  Provide  boilerplate  and  “fill  in  the  blank”  RFP  and  contract  language 

-  Recommend  ways  to  apply  guidance  and  language  to  align  THEIR 
acquisition/procurement  documentation  with  a  program  Life-cycle  and  DoD 
Enterprise  view 

Forms 

-  M&S  Acquisition  Guide.  8-10  pages  providing  questions,  measures,  langauge 
and  application  principles  for  their  consideration 

•  Managed  AT&L  publication? 

-  Influence.  DoD  and  component  acquisition  and  procurement  regulations, 
policies  and  guidance. 

•  FAR;  DFARS;  DoDD  5000.1;  DoDI  5000.2;  DoD  component  policy  and  guidance 

•  Defense  Acquisition  Guidebook 

•  Guide  for  Integrating  Systems  Engineering  into  DoD  Acquisition  Contracts 

-  Continuous  Improvement.  Establish  a  process  to  collect  lessons  learned,  update 
M&S  Acquisition  Guide  and  target  reviews  and  updates  of  DoD  regs,  policies  and 
guidance 
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Modeling  and  Simulation 


MODEL:  The  knowledge  and 

understanding  that  the  scientist 
has  about  the  world  is  often 
represented  in  the  form  of 
models.  ...  A  model  is  a 
representation  containing  the 
essential  structure  of  some 
object  or  event  in  the  real 
world.  -  From  Introductory 
Statistics:  Concepts,  Models, 
and  Applications  by  David  W. 
Stockburger 


predict 

understand 

evaluate 


executable 

Knowledge 


MODEL:  A  representation  of  a  system 

that  allows  for  investigation  of  the 
properties  of  the  system  and,  in  some 
cases,  prediction  of  future  outcomes. 
Models  are  often  used  in  quantitative 
analysis  and  technical  analysis,  and 
sometimes  also  used  in  fundamental 
analysis.  -  From  "InvestorWords" 


SIMULATION:  The  process  of 

designing  a  model  of  a  real 
system  and  conducting 
experiments  with  this  model 
for  the  purpose  either  of 
understanding  the  behavior  of 
the  system  or  of  evaluating 
various  strategies  (within 
the  limits  imposed  by  a 
criterion  or  set  of  criteria) 
for  the  operation  of  the 
system."  from  Introduction  to 
the  Art  and  Science  of 
Simulation;  1998  Winter 
Simulation  Conference  1998. 
R.E.  Shannon. 


a  way  of  thinking  and 
reasoning  about  systems 


The  Essence  of  Intellectual  Property 


Essential 

structure 
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Buyer 


Value  Perspectives 
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Other  Considerations  for  M&S  Acquisition 


•  Have  a  Rigorous  M&S  Planning  Process 

•  Have  an  DoD  Enterprise/Life-cycle  Perspective 

-  Discovery,  Access,  Interoperability,  Reuse 

•  Have  a  VV&A  plan 

•  Seek  to  leverage  the  best  use  of  M&S  capabilities  from  industry  and  government 

-  Avoid  building  new  capability  when  existing  assets  can  be  used 

-  Strongly  consider  COTS  solutions;  resist  temptation  to  favor  custom  solutions 

•  Seek  to  maximize  value  for  the  government 

-  Understand  market  value  of  proposed  IP 

-  Maximize  bargaining  leverage  by  identifying  requirements  in  the  competition  phase. 

-  Be  explicit  about  minimizing  risk  of  exposure  of  IP 

•  Understand  time-dependence  of  need 

•  Consider  DoD  Enterprise  and  Life-cycle  Issues 

-  M&S  Strategy  /  Vision 

-  Reuse  and  interoperability  standards. 

-  Assure  M&S  and  associated  data  are  available  for  other  users  throughout  the  system’s  life- 
cycle. 

•  Protect  proprietary  and  intellectual  property  rights  of  M&S  developers. 
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Statute,  Regulation,  Policy  (e.g.) 

•  Title  5  U.S.  Code  Section  552;  “Public  information;  agency  rules,  opinions,  orders,  records,  and  proceedings”  -- 
FOIA 

•  Title  10  U.S.  Code  Chapter  137;  “Procurement  Generally” 

•  Title  10  U.S.  Code  Chapter  140;  “Procurement  of  Commercial  Items” 

•  Title  10  U.S.  Code  Chapter  141;  “Miscellaneous  Procurement  Provisions” 

•  Title  18  U.S.  Code;  “Crimes  and  Criminal  Procedures”  -  Trade  Secrets  Act,  Economic  Espionage  Act 

•  Title  15  U.S.  Code;  “Commerce  and  Trade” 

Title  1 7  U.S.  Code;  “Copyrights” 

•  Title  35  U.S.  Code;  “Patents” 

•  Title  41  U.S.  Code;  “Public  Contracts” 

•  Uniform  Trade  Secrets  Act  (compilation  of  State  laws) 

•  Federal  Acquisition  Regulation  (FAR) 

•  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS) 

•  International  Traffic  in  Arms  Regulations  (ITAR) 

•  DoD  Instruction  5000.2,  “Operation  of  the  Defense  Acquisition  System,”  May  12,  2003. 

•  DoD  Directive  8000.1 ,  “Management  of  DoD  Information  Resources  and  Technology,”  February  27,  2002. 

•  DoD  Directive  8320.2,  “"Data  Sharing  in  a  Net-Centric  Department  of  Defense",  December  2,  2004. 

•  DoD  Instruction  5000.61,  "DoD  Modeling  and  Simulation  (M&S)  Verification,  Validation,  and  Accreditation 
(VV&A),"  May  13,  2003. 

•  SECNAV  Instruction  5200.40,  “Verification,  Validation  and  Accreditation  (VV&A)  of  Models  and  Simulations,”  April 
19,  1999. 

•  COMOPTEVFORINST  5000.1 ,  “Use  of  Modeling  and  Simulation  (M&S)  in  Operational  Test  (OT),”  September  9, 
2004. 

•  MIL-HDBK-245D,  “DoD  Handbook  for  Preparation  of  Statement  of  Work,”  April  3,  1996. 

•  MIL-HDBK-61  A(SE),  “Configuration  Management  Guidance,”  February  7,  2001 . 

•  DoD  VV&A  Recommended  Practices  Guide,  2004. 

•  DoD  5000. 59-M,  “DoD  Modeling  and  Simulation  Glossary”,  Jan  1998. 
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A  Planned,  DoD  Enterprise/Life-cycle 

Commitment 


Modeling  and  Simulation  Planning  Process 
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Concept  Refinement 
Trade  Space  Analyses 
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Acquisition  Strategy 


Develop  Modeling  and  Simulation  Support 

* 

Plan  (M5SF) 

R 

1 

Update  USSR 


Technology  Development  Trade  Space 
Analyses 


System  Development  &  Demonstration  Trade  Space  Analysis 


Operations  and  Support  Trade  Space  Analyses 


Develop  W&A  Plan 


Develop  T&E  Strategy 


*  Develop  TEMP 


Identify  MSS  Test 
Assets 


Verification  &  Validation 
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Operations  and  Sustainment 


PBL  Implementation 


KD 


RFP  and  Contracting  Guidance  has  Little  Value  Without  an  M&S  Plan 
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Proposed  Considerations  for  RFP 


•  What  New  Modeling  and  Simulation  Capabilities  are  Required? 

-  How  will  the  capabilities  be  used? 

•  Can  the  details  of  the  requirement  be  clearly  articulated  in  the  RFP? 

•  Are  there  existing  assets,  currently  available  to  the  government,  that 
can  be  applied  to  all  or  part  of  the  requirement? 

-  Appropriate  pedigree 

-  Appropriate  for  use 

•  What  are  the  applicable  model  parameters,  standards,  interfaces, 
interoperability,  and  output  requirements? 

-  Alignment  with  Program  and  DoD  Enterprise  Goals 

•  What  level  of  rights  to  the  input/output  data  be  required? 

-  Access,  Publish  and  Modify 

-  Consider  fixed-price  options  to  purchase  data  rights  downstream 
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Proposed  Source  Selection 
Considerations 


•  Technical 

-  Feasibility  of  Proposed  Solution 

-  Availability  of  Required  Data 

•  Value 

-  Cost  v.  Benefit 

-  Risk 

•  Spirit  and  Letter  of  the  RFP 

-  Meets  Requirements 

-  Meet  Demands  of  Program  and  DoD 
Enterprise  Strategy 

-  Fitness  for  Use 

-  Pedigree 

-  Adherence  to  standards  or  provide 
reasonable  argument  for  exemption 

•  Life-cycle  Implications 

-  Use  Costs 

•  Operations 

•  Maintenance 

•  License  fees 

-  Continued  Support  -  Commitment  to 
Sole  Source? 


•  Offeror  Qualifications 

-  Documented  systems-engineering 
process 

-  Existing  information  sharing 
infrastructure 

-  Successful  experience  using  a  wide  of 
models  and  simulations 

-  Successful  application  and 
demonstration  of  M&S  interoperability 
and  reuse  standards 

-  Documented  VV&A  process 

-  Staff  with  documented  M&S  expertise 

•  Data  Rights 

-  Access  to  required  input  and  output 
data. 

-  Rights  to  use,  publish  and  modify  data 
as  required 

-  Are  there  fixed-price  options  for 
purchasing  data  rights  later  in  the 
acquisition  process? 
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Example  Contract  Language  / 
Standard  Clauses 

DFARSt 

•  Rights  in  Technical  Data  -  Non-Commercial  (from  DFARS) 

•  Quality  Assurance  (from  clauses  to  sub-contractors  to  Boeing 

working  NASA  ISS) 

•  Software  Management  (from  FCS) 

•  Data  Formats:  Data  Content;  Data  Procedures  (from  U.S.  Army 

Corps  of  Engineers  -  CADD) 

•  Offeror  Qualifications  (from  Space  Based  Radar  Task  Order) 

•  NAVSEA  SEAPORT-e 

-  Modeling,  Simulation,  Stimulation,  and  Analysis  Support 

-  This  functional  area  consists  of  the  application  of  a  standardized,  rigorous,  structured 
methodology  to  create  and  validate  a  physical,  mathematical,  or  otherwise  logical 
representation  of  a  system,  entity,  phenomenon,  or  process.  The  functional  area  involves  the 
use  of  models,  including  emulators,  prototypes,  simulators,  and  stimulators,  either  statically 
or  over  time,  to  develop  data  as  a  basis  for  making  managerial,  technical,  strategic,  or 
tactical  decisions. 
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Summary 

•  Have  a  Rigorous  M&S  Planning  Process  and  Documented  Plan 

-  VV&A  Plan 

-  Configuration  Management  Plan 

-  Reference  document  in  RFP  and  Contract 

•  Have  an  DoD  Enterprise/Life-cycle  Perspective 

-  Discovery,  Access,  Interoperability,  Reuse 

•  Next  Steps 

-  Continue  to  refine  guidance/considerations 

-  Review  existing  RFPs,  Contracts,  etc 

-  Socialize  and  with  Industry  and  Government  Players 

•  Technical  Personnel 

•  Contracts  Personnel 

•  Legal  Personnel 

-  Publish  M&S  Acquisition  Guide 

-  Develop  plan  to  influence  DoD  and  component  regulations,  policies  and 
guidance 

-  Establish  Continuous  Improvement  Process 
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Example  Proposed  Contracting 
Language  Considerations 

The  Contractor  Shall: 

•  Develop,  update  and  maintain  M&S  capability  in  compliance  to  a 
documented  configuration  management  process. 

•  Describe  his  use  of  M&S  throughout  the  system’s  lifecycle  as  an  integral 
part  of  the  Systems  Engineering  Plan. 

•  Develop,  implement,  and  update  M&S  planning  describing  the  contractor’s 
use  of  M&S  in  the  development  of  the  system  as  an  integral  part  of  the 
Systems  Engineering  Plan. 

•  Collaborate  with  the  government  to  develop  a  system  model  using 
architectural  modeling  tools  in  order  to  achieve  mutual  understanding  of  the 
system  under  development. 

•  Perform  VV&A  of  modeling  and  simulation  capabilities  consistent  with  risk 
and  DoD  M&S  VV&A  Instruction  5000.61. 

•  Develop,  and  make  accessible  to  the  government  an  virtual  representation 
of  its  system  in  the  form  of  a  model  or  simulation. 

•  Include  in  its  participation  in  the  required  program  reviews,  performance  of 
the  system  under  development  using  simulations  demonstrations  with  M&S 
developed  by  the  contractor. 
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Modeling  and  Simulation 
Resource  Reuse 
Business  Model 


Dennis  P.  Shea 
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Outline 


Problem  statement 

-  Barriers  to  reuse 

-  Repositories  as  necessary  but  insufficient  incentives 

-  On  the  need  for  an  M&S  resource  reuse  business  model 

Framework  of  a  business  model 
Focus  areas 

-  Laws  and  policies  affecting  intellectual  property 

-  Extending  Tech  Data  rights  to  M&S  resources 

-  Mechanisms  to  support  intra-government  transactions 

-  Measures  to  encourage  open-source  software 

-  Improved  discovery  and  distribution  tools 


The  Problem: 

Inefficient  Use  of  M&S  Resources 
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Few  M&S  resources  are  reused  -  either  during  a  single 
program’s  lifecycle  or  across  acquisition  programs. 


Tools 

Data 

Environment 

Models 

Input  datasets 

Architectures 

Network  resources 

Simulations 

Scenarios 

Interfaces 

SME  expertise 

Federations 

Threat  data 
Algorithms 

Protocols 

Utilities  (post¬ 
processors) 

Environmental 

info 

VV&A  templates 

Absence  of  incentives  for  M&S  managers 
and  developers  is  a  contributing  factor. 


Reusable  M&S  resources  generally 

contain  valuable  intellectual  property 
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•  Intellectual  property  refers  to  creations  of  the  mind :  inventions,  literary 
and  artistic  works,  and  symbols,  names,  and  images  used  in  commerce. 

-  In  M&S  the  IP  is  often  encapsulated  in  the  source  code  and  data  sets 

•  DOD’s  access  to  M&S  IP  developed  under  contract  is  governed  by  both 
copyright  law,  patent  law,  and  the  procurement  regulations  contained  in 
the  DFARS 

-  These  laws  affect  the  Government's  ability  to  use,  reproduce,  modify, 
and  release  the  resource  to  one  or  more  potential  users 

•  Control  of  IP  is  determined,  in  part,  by  who  funded  development 

-  Government,  Industry,  or  Mixed 

-  But  formal  title  is  generally  retained  by  the  contractor-developer 
regardless  of  funding  source 

-  DoD  acquisitions  that  involve  a  mix  of  government  and  IRAD  funded 
technologies  pose  a  challenge  in  determining  control  “rights” 
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Barriers  to  M&S  Resource  Reuse 


Users  lack  awareness  of 
reusable  resources 

Insufficient  details  about 
reusable  resources 

Hard  to  assess  the  true 
capabilities  and  limitations 
of  existing  resources 

Resources  not  in  a  form 
suitable  for  reuse 

Users  lack  trust  in 
resources  developed  by 
others/  NIH 

Model  is  available  but  not 
the  data 

M&S  components  don’t 
work  well  together 


•  Repositories  are  incomplete 
and  not  current 

•  Little  insight  into  how 
resources  have  been  used  in 
the  past,  including 
successfully  and  failures 

•  Difficult  to  access  the  actual 
resource 

•  Difficult  to  adapt  existing 
resources  to  new  problems 

•  No  mechanism  to 
compensate  developer  for 
resource  investment  and 
guidance  on  use 

•  No  mechanism  to  protect 
developer  from  mischievous 
uses 
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M&S  repositories,  registries,  etc. 
can  be  critical  enablers 


[A 


If  they  ... 

*  Identify  items  that  conform  to  quality  standards  and  list 
pedigree 

*  Use  a  comprehensive,  consistent,  and  well-structured 
taxonomy 

*  Include  revisions,  maintenance,  and  needed  taxonomy 
updates 

*  Provide  meta-data  which  includes  semantics  descriptors 

*  Include  user-friendly  search  methods 

*  Are  adequately  funded  /  supported 

-  including  staff  to  match  users  with  resources 

*  Require  security  to  control  access,  but  not  onerous  security 

*  Include  motivation  to  resource  providers  to  populate  and 
update 

However,  ...  6 
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Improved  M&S  repositories 
could  overcome  some  barriers 


Users  lack  awareness  of 
reusable  resources 

Insufficient  details  about 
reusable  resources 

Hard  to  assess  the  true 
capabilities  and  limitations 
of  existing  resources 

Resources  not  in  a  form 
suitable  for  reuse 

Users  lack  trust  in  resources 
developed  by  others/  NIH 

Model  is  available  but  not  the 
data 

M&S  components  don’t  work 
well  together 


•  Repositories  are  incomplete 
and  not  current 

•  Little  insight  into  how 
resources  have  been  used  in 
the  past,  including 
successfully  and  failures 

•  Difficult  to  access  the  actual 
resource 

•  Difficult  to  adapt  existing 
resources  to  new  problems 

•  No  mechanism  to  compensate 
developer  for  resource 
investment  and  guidance  on 
use 

•  No  mechanism  to  protect 
developer  from  mischievous 
uses 
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But  repositories  alone  are 

insufficient  to  motivate  reuse: 


*  Without  incentives  to  populate,  repositories  will 
not  include  a  comprehensive  set  of  available 
resources 

*  Existing  resources  require  additional  work  to 
adapt  to  new  problems  and  support  to  guide  their 
application 

*  Repositories  often  don’t  facilitate  the  transaction 
to  obtain  the  actual  resource 

*  Repositories  don’t  protect  the  original  developer 
from  resource  misuse  by  new  users 
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Project  objective 


Develop  an  economic  business  model  that  will 
make  the  reuse  of  M&S  resources  an  attractive 
option  for  both  consumers  and  providers  of  these 
resources 

-  Identify  incentives  for  consumers  and 
providers 

-  Establish  how  the  transactions  will  occur 

-  Identify  funding  and  procurement  policies  and 
legislation  that  will  need  to  be  changed  to  put 
the  reuse  model  into  practice 


Project  results 


A  business  model  would  help  to 

*  Promote  competition  in  the  market  for  M&S 
assets  by  allowing  for  appropriate  access  to 
resources  owned  or  controlled  by  DoD 

*  Ensure  that  the  producers  of  M&S  assets 
receive  a  fair  return  on  their  investments 

*  Create  an  IT  environment  that  breaks  down 
barriers  to  collaboration,  teamwork,  NIH, ...  and 
encourages  a  longer-term  planning  horizon. 
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Business  Model  Elements 


A  business  model  would  describe  the 

•  Value  to  M&S  consumers  produced  by  the  ability 
to  access  and  reuse  M&S  resources; 

•  The  reciprocal  value  to  M&S  producers  through 
transactions  that  result  in  the  reuse  of  their 
resources; 

•  The  capabilities,  partners,  and  business 
processes  required  to  create  and  deliver  this 
value; 

•  The  motivation,  compensation  principles,  and 
policy  necessary  to  sustain  a  mutually  beneficial 
relationship  between  these  entities 
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M&S  Resource  Reuse  Business  Model 


IP  Suppliers  & 

Support 

Infrastructure 
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Value  activities 

-  Develop 

-  Test 

-  Validate 

-  Prototype 


Core  capabilities 

-  H/W  &  S/W 

-  System  information 

-  Org  &  Op  Knowledge 

-  Conceptual  models 


Partner  network 

-  Gov’t  agencies 

-  Labs 

-  Industry 

-  International 


Customer 
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Value 

Proposition 

-  Savings  (time/$$) 

-  Authoritative 

-  Joint  context 

-  Interoperability 


Target  Mkt 

-  PEOs,  PMs 

-  Dir  Training 

-  Hd  Analysis 

-  Service/Component 


Customer 

Relationships 

-  Discovery  tools 

-  Trust/  MOUs 
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Distributic 

-  Access  coi 

-  IP  Intermed 
-MOUs 

m  channel 

itrol 

liaries 

Compensation 

-  Licensing 

-  Royalties 

-  Support  $$ 

-  Purchase  options 
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M&S  Resource  Transactions 


Discovery/Distribution  Mechanism 
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Gov’t  Agency  2 


Provide 

resource 


Complete 

agreement 


M&S  Team 

In-house,  Lab, 
Industry 


Provide  support  to  adapt  and  apply  resource 


Payment  for  support  services  received 


M&S  Team 

In-house,  Lab, 
Industry 
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Study  findings  could  include 
recommendations  on: 


•  Extending  Tech  Data  rights  to  M&S 
resources 

•  Mechanisms  to  support  intra-  government 
transactions 

•  Measures  to  encourage  open-source 
software 

•  Improved  discovery  and  distribution 
tools 
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John  Warner  National  Defense 
Authorization  Act  for  FY2007 


Sec.  802.  Additional  Requirements  Relating  to  Tech  Data  Rights 


•  Section  2320  of  10  U.S.C.  is  amended  by  adding  the  following  new  subsection: 

(e)  The  Secretary  of  Defense  shall  require  program  managers  for  major  weapon 
systems  and  subsystems  of  major  weapon  systems  to  assess  the  long-term 
technical  data  needs  of  such  systems  and  subsystems  and  establish 
corresponding  acquisition  strategies  that  provide  for  technical  data  rights  ... 

•  Motivated  by  a  need  to  develop  alternatives  to  sustain  systems  over  their 
lifecycle,  including 

-  Shifting  maintenance  to  DoD  and 

-  Re-competing  contracts  for  spare  parts 

•  Could  be  extended  to  include  models ,  simulations ,  and  associated  databases 
developed  and/or  used  in  training ,  acquisition ,  testing,  evaluation,  and  analysis 
of  weapons  systems 

•  Motivation  is  a  desire  to  support  upgrades  to  weapon  systems 


Repositories  like  SHARE,  originally  established  to  contain  computer  software 
and  tech  data  associated  with  combat  weapon  system,  could  be  extended  to 
include  M&S  software  and  associated  data 
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Federal  Agencies  as  IP  Intermediaries 


•  NTIS  as  intermediary  for  IT  and 
government  publications 

•  NIH  Office  of  Technology  Transfer 
(OTT)  as  intermediary  for  IP 
developed  by  staff  and/or  grantees 

•  NASA  Open  Source  Initiative  as  a 
distribution  vehicle  for  software 

•  NASA  IPP  as  intermediary  for  IP 
developed  by  staff  and/or  grantees 
(includes  royalty  opportunities  for 
innovators) 


Examples  of  Gov-2-Gov  Agreements 


•  Inter-Governmental  Service  Agreements 
(IGSAs) 

-  Immigration  and  Customs  Enforcement  (DHS/ICE) 
leases  space  in  over  600  state  and  local  jails  and 
prisons  in  order  to  house  alien  detainees;  the 
facilities  used  include  both  government-owned  and 
contractor-owned  institutions. 

•  Interagency  Agreements  (IAA) 

-  Federal  Occupational  Health  (DHHS/FOH)  negotiates 
lAAs  with  federal  agencies  to  provide  on-site  health 
care;  services  are  provided  by  health  care 
professionals  working  for  FOH  as  independent 
contractors. 
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Proprietary,  non-commercial 
m  software  poses  obstacles  to  reuse 

*  Development  environment  restricted  to  internal 
ideas  and  internal  technology 

-  No  reuse  of  the  work  from  other  organizations 

*  Often  require  NDA  to  look  under  the  hood 

-  NDA  governs  disclosure  of  models,  source  code, 
documentation,  databases,  drawings,  know-how, 
formulas,  processes,  concepts,  flow  charts,  designs, 
ideas,  technical  plans, ... 

-  Prohibits  copying,  altering,  modifying,  disassembly, 
reverse  engineering,  or  decompiling  the  resource 

*  Can’t  transfer  asset  to  a  third  party  for  reuse 

However,  the  absence  of  competition  and 
potential  for  collaboration  may  lead  to  monopoly 
markets  in  niche  areas  such  as  DIME/PMESII  18 


Open  Source  Software 
supports  reuse 


•  OSS  breaks  through  several  barriers  preventing  reuse: 

-  Not  ready  for  reuse  “Not  invented  here” 

-  Components  don’t  play  nicely 

-  Poor  capability  assessment 

-  Insufficient  detail  &  documentation 

-  Difficult  to  access  resources 

-  Little  info  regarding  past  use 

-  Incomplete  and  not  current  repositories 

-  Resources  hard  to  adapt 

•  Costs  of  OSS  lower  (transition,  lifetime,  support,  upgrades) 


•  Open  standards  used  in  OSS  support  interoperability,  and 
hence  reuse 


•  Reusables  tend  to  be  abstract  and  harder  to  engineer.  OSS 
patterns,  which  are  recurring  solutions  to  common  software 
problems,  help  alleviate  this  and  other  reuse  impediments. 
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Improved  Discovery  tools:  Bringing 
.  M&S  developers  and  users  together 


Catalog  Access  to 
IP  Assets 


Direct  IP  Distribution 


3rd  Party/ 

External 

Consolidator 

IP  Search  Engines 

IP  Exchange  Platforms 

Publishers 

Resellers 

iTunes 

Library  of  Congress 

Copyright  Collectives 

Non-Profit/ 

Library  Consortia 

Shareware,  Open  Source 

Government/ 

Fed.  Tech  Transfer 

Software  Dist. 

Self- 

Sites 

Federal  Data  Sources 

Managed 

Univ.  Tech  Transfer 

Non-Profit  Organizations 

Sites 

Software  Escrow  Agents 

External  Consolidator,  IP  Catalog  Access 


IP  Search  Engines 

-  Cambia  Patent  Lens 

-  Delphion  Research 

-  Google  Patents 

-  PatentCafe 

-  PIPRA* 

-  Thompson  Dialog 

-  Thomson 
MicroPatent 

-  Thomson  Pharma 

-  WIPO  Digital  Patent 
Library* 


iTunes 

IP  Exchange 
Platforms 

-  BirchBob 

-  Idea  Trade  Network 

-  MVS  Solutions 

-  PharmaTransfer 

-  TechEx 

-  Yet2 


(^nonprofit  organization) 
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External  Consolidator,  IP  Distributors 


•  Publishers 

-  IEEE  Xplore 

-  Elsevier 

-  Nat’l  Acad.  Press 

-  Lexis,  Westlaw 

-  Newspaper  web 
sites 


•  iTunes 

•  Resellers  of 
Copyrighted 
Material 

-  JSTOR 

-  EBSCOHost 

-  Ingenta/Uncover 
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Self-Managed,  IP  Catalog  Access 


•  Library  of  Congress 
On-Line  Catalog 

*  Library  Consortia 

-  Membership  (ex: 
On-Line  Computer 
Library  Center, 
oclc.org ) 

-  Open  access  (ex: 
Washington 
Research  Library 
Consortium, 
wrlc.org) 


•  Fed  Tech  Xfer 

-  NASA 

-  NIH 

-  NIST 

-  DoD,  including  lACs 

-  Other  Fed  CRADA 

•  Univ.  Tech  Xfer 

-  See  Association  of 
University  Technology 
Managers  (autm.net )  for 
background 
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Self-Managed  IP  Distributors 


[A 


Copyright  collectives: 

*  ASCAP,  BMI  (in  US) 

*  ALCS  (in  UK) 

*  J  ASA  RAC  (in  Japan) 
Shareware  Distributors 

*  Tucows,  tucows.com 

*  PC  Magazine/  Digital 
River,  www.regnow.com 

Open  Source  Software  Dist. 

*  ossid.org  (background) 

*  opensource.org 

*  redhat.org 

*  NASA  Open  Source 
Initiative, 

opensource.arc.nasa.gov 


Federal  Data  Sources 

*  gpo.gov,  thomas.loc.gov 

*  Bureau  of  the  Census 

*  Bureau  of  Labor  Statistics 

*  National  Weather  Service 

*  Department  of  Energy 

Non-profits 

*  FFRDCs  (sei.cmu.edu, 
CNA,  RAND,  IDA) 

*  “Think  Tanks”  (AEI, 
Brookings,  Cato,  Urban 
Institute,  etc.) 

*  Constituency 
Organizations  (Ex:  eff.org, 
aarp.org) 

Escrow  Agents 

*  Example:  innovasafe.com 


Summary:  A  business  model  takes  reuse 


Field  of  Dreams,  To  Government  Contracting,  To  Business  Viability 


Build  it  and  they  will  come... 


Operational 

- \ 

Perfoimance 

Requirements 

Specs 

(Yfafsfe  Gats)  t 

I  Many  frovideta 
1  (OFi  Smell  BLEfeea, 
taadenla,  Inifilifl 


The  Open  Architecture  Business  Model 


Make  it  work  w/n  DoD  contracting... 


Make  it  profitable  for  participants... 


Backup 


Software  Copyright  Overview 


Open  Source 


Free 


- \ 

ProprieiLiry 


Dovviiluud 


Closed 


Shareware 


Free  SofixCarc 


r 

j  Public  domain 

W  ■*  ^  J 
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XFrccSft  Style  ] 
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- 

Copylefted 

f 

K 
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Exactly  What  is  Open  Source 
Software? 


THE  Open  Source  (partial)  Definition: 

•  Free  redistribution  (no  royalty  or  fee) 

•  Source  code  must  be  provided  or  made  otherwise  freely 
available,  upon  redistribution 

•  Source  code  can  be  modified.  Derived  works  can  be 
redistributed  under  terms  of  original  license 

•  Same  rights  provided  to  all  recipients 

•  No  restriction  on  other  software  bundled  with  the  OSS 

Source  code  distribution  is  only  required  by 
GNU’s  General  Public  License  (GPL)  and  Lesser 
GPL,  and  then  only  upon  distribution  of  work 
derived  from  the  original  source. 

-  So,  OSS  ok  for  sensitive/classified  software! 
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Open  Source  Software  in  the 
Private  Sector 


OSS  is  found  throughout  the  private  sector: 

-  Apache  (web  server)  ,  Linux,  Java 

-  MySQL  (database) 

-  Other  middleware  and  internet  infrastructure 

-  Supercomputing  clusters  in  academia  (Virginia  Tech,  Penn  State, 
Georgia  Tech,  etc.)  running  OSS 

-  Parallel  high-performance  computing  tools:  OpenMP,  Open  MPI, 
MPICH 

Delta3D,  Genesis3D,  &  Irrlicht  engines,  for  3D  simulations  and 
games 

IBM  invested  over  $1B  in  Linux  development  and  promotion 

Sun  developed  Star  Office  to  compete  with  MS  Office  and  then 
opened  the  source.  The  result  is  Open  Office,  which  is  MS 
Office  compatible  and  uses  the  Open  Document  format  also. 

Several  other  companies  have  sponsored  or  contributed  to 
OSS:  Boeing,  Northrop  Grumman,  Novell 


Open  Source  in  DoD 
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•  US  Army  is  single  largest  install  base  of  Red  Hat  Linux 

•  Open  Source  Software  Image  Map  (OSSIM)  project 

-  High  performance  software  system  for  remote  sensing,  image 
processing,  geographical  information  systems  and 
photogrammetry. 

-  OSSIM  (pronounced  “Awesome”)  currently  deployed  for 
commercial  and  government  use...  e.g.  provides  imagery  to 
NASA  and  Google  Earth,  embedded  in  several  classified  and 
sensitive  solutions  in  intelligence  community. 

•  OneSAF:  A  simulation,  training  and  instrumentation 
environment 

-  Used  by  US  Army  and  USMC  (embedded  simulation  engine 
for  US  Army  Future  Combat  Systems) 

-  Virtually  unlimited  in  creating  virtual  military  environments. 

•  OpenEaagles:  USAF  simulation  environment 

•  SubrScene:  real-time  simulation  visualization  toolkit. 

-  Used  by  USAF  for  emersion  in  virtual  environments  for  3D  fly- 
through 
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Other  Federal  Government  Open 
Source  Software  initiatives 


NASA  -  Has  released  over  20  open  source  apps 

-  CosmosCode  initiative:  enlists  volunteers  to  develop  code  for 
space  missions 

-  Supercomputer  (one  of  world’s  most  powerful)  uses  SGI  Linux 

Navy  Open  Architecture  -  actively  encourages  open  source 
and  standards  in  acquisitions 

FAA  saved  $15M  and  12  months  switching  to  Red  Hat  Linux 
(from  Unix)  on  air  traffic  control  computers 

USAF,  National  Oceanographic  and  Atmospheric  Admin, 
use  SGI  Linux. 

Army  Future  Combat  System  -  uses  Linux  and  open  source 
simulation  software 

Defense  Medical  Logistics  Standard  Support 

-  Apache,  OpenSSL,  ModSSL,  Stunnel 

Navy  -  Total  Ship  Computing  Environment  Initiative  (TSCEI) 
runs  on  Red  Hat  Linux 

-  Will  support  carriers,  amphibious  ships,  destroyers, ...  31 
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The  Use  of  Enterprise  Architecture 
to  Support  the  Development  of  the 
Next  Generation  Air  Transportation 

System  (NextGen) 
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Outline 


•  Introduction 

•  Background 

•  Approach 

•  Lessons  Learned 

•  Summary 
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MITRE 


Baseline  Values 


Why  NextGen? 


Shift  in  passengers  per  flight 
(e.g.,  A380,  reverse  RJ  trend, 
vhiqher  load  factor) _ 


2014  and  later  Baseline  analysis 
will  use  OEP  &  FACT  fapacities 


pp 


Passengers 


1.8-2.4X 


Biz  shift 

•  Smaller  aircraft,  more  airports 


Increase  of  over  10 
passengers  per  flight 


~3X 

Note:  Not  to  scale 


Flights 


Biz  shift 

•  2%  shift  to  micro  jets 


2014 


2025 


20?? 


2004  Baseline 

•  Current  Flight  Schedule 
^  •  Current  Capacities 


Growth  in  volume  and 
complexity  of  operations 
Scalable  to  encompass  a 
range  of  possible  futures 
Broader  diversity  of: 

-  Aircraft  performance 
characteristics 

-  Aircraft  capabilities 

-  Operator  business 
models 

Space  Operations 
Unmanned  Aerial  Systems 


Transformation  is  Needed  to  Accommodate 
Projected  Traffic  Levels  and  Characteristics 


MITRE 


Vision  100  Legislation  - 
JPDO  Responsibilities 


•  Create  and  carry  out  an  integrated  plan  to  achieve  NextGen 

•  Coordinate  goals  and  priorities  for  research  and 
development 

•  Coordinate  research  activities  within  the  Federal 
Government  and  across  United  States  aviation  and 
aeronautical  firms 

•  Facilitate  the  transfer  of  technology  from  research 
programs  to  Federal  agencies  with  operational 
responsibilities  and  to  the  private  sector 

•  Create  a  transition  plan  for  the  implementation  of  NextGen 

>  Operate  in  conjunction  with  relevant  programs 
in  specified  government  agencies 

>  Consult  with  the  public  and  ensure  the  participation  of 
experts  from  the  private  sector 
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— 


Scope  of  NextGen 


Next  Generation  Air  Transportation  System  (NextGen) 


*  Envisioned  2025  system  and  transition  from  today 

*  Satisfy  capacity,  efficiency,  safety,  defense,  and 
security  needs 

-  Responsive  to  environmental  concerns 

-  Responsive  to  global  harmonization  requirements 

*  Planned  jointly  by  public  and  private  sector 
stakeholders  through  JPDO 

-  Civil  aviation,  National  Defense,  Homeland  Security, 
and  Transiting  Spaceflight 
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What  is  NextGen? 


Transformation  goals: 

•  Leadership  in  global  aviation 

•  Scalable  up  to  3x  increase  in  capacity 

•  Ensure  our  national  defense  (readiness 
and  homeland  security) 

•  Enhance  the  environment  (noise,  air 
quality) 

•  Improve  safety 

•  Globally  harmonized 


Capabilities: 

Network-Enabled  Information  Access 
Performance  Based  Operations  &  Services 
Weather  Assimilated  into  Decision  Making 
Layered,  Adaptive  Security 
Position,  Navigation,  and  Timing  Services 
Aircraft  Trajectory  Based  Operations 
Equivalent  Visual  Operations 
Super  Density  Arrival/Departure  Operations 
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Organization  Developing  the  NextGen 


Organization  Developing  the  Architecture 

The  NextGen  Enterprise  Architecture  is  developed  and  maintained  by  the  Enterprise  Architecture  and  Engineering 
Division  (EAED)  of  the  Joint  Planning  Development  Office  [J P DO),  an  Executive  Office  created  to  support  and 
coordinate  the  efforts  of  Government  and  Private  organizations  in  the  development  of  the  Next  Generation  Air 
Transportation  System  as  envisioned  by  the  President's  "Vision  100"  operational  concept, 
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Source:  “Joint  Planning  and  Development  Office  -  Program  Management  Plan  VI  .0  for  the 
Enterprise  Architecture  and  Engineering  Division”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 


MITRE 


— 


Roles  and  Responsibilities 


•  Government  Leadership 

-  EAED  -  Division  Director  -  Mr.  Ed  Waggoner 

•  Directs  all  EAED  activities  and  in  charge  of  all  staff  that  support 
EAED 

•  Represents  the  EAED  on  the  JPDO  Integration  Council 

-  EAED  -  Chief  Architect  -  Mr.  Jay  Merkle 

•  Responsible  for  leading  the  development,  coordination, 
collaboration,  and  validation  of  the  NextGen  ConOps,  EA  and  IWP. 

•  Booz  Allen  &  Hamilton:  JPDO  -  EAED  primary  support  contractor, 
primary  responsibilities: 

-  Provide  support  and  guide  the  development  of  the  EAED  concepts  and 
development  practices 

-  Identify  architecture-related  issues  and  actions  that  need  to  be  addressed 

-  Develop  and  review  architectural-related  products 

-  Manage  and  implement  approved  architectural  changes 

•  MITRE:  JPDO  EAED  FFRDC,  primary  responsibilities: 

-  Provide  guidance  and  recommendations  to  inform  the  development  of 
engineering  products. 

-  Perform  independent  technical  review  and  validation  of  major  JPDO  engineering 
products. 

-  Help  ensure  products  are  appropriate  in  terms  of  content  and  structure  and  will 
identify  integration  needs,  dependencies  and  areas  requiring  alignment  among 

the  products 
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Approach 


MITRE 


System-Wide  Transformatio 


NextGen  Requires  Innovation  Across  All  Lines  of  Development 


Policy 
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Source:  Briefing  titled  “The  Next  Generation  Air  Transportation  System  -  AAITPF 
EAED  Products  Status”  by  Dr.  Edgar  Waggoner,  dated  16  May  2007. 


NextGen  Planning  Products 
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Source:  Briefing  titled  “NextGen  101  -  Next  Generation  Air  Transportation  and  the  Joint 
Planning  and  Development  Office  Orientation”,  by  MITRE,  dated  June  2007. 


MITRE 
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Source:  Briefing  titled  “The  Next  Generation  Air  Transportation  System  - 
EAED  Products  Status”  by  Dr.  Edgar  Waggoner,  dated  16  May  2007. 


MITRE 


Key  NextGen  Products  from 
Enterprise  Architecture  and 

Engineering 


NextGen  EA  Framework 


•  Is  a  combination  of  the 
Federal  Enterprise 
Architecture  (FEA)  and 
DoD  Architecture 
Frameworks  (DoDAF), 
products  and  processes 

•  The  DoDAF  OV5  along 
with  the  FEA  business 
reference  model  (BRM) 
act  as  the  integration 
mechanism  of  the  two 
frameworks 


•  Establishes  a  common 
lexicon  and  creates  a 
starting  point  for  partner 
agencies  to  align  and 
“drill-down”  their 
respective  architectures 

Source:  Briefing  titled  “NextGen  Enterprise  Architecture  (EA) 

Focus:  Federal  Enterprise  Architecture  (FEA)  &  Federal 

Transition  Framework  (FTF)  Initiatives”  by  JPDO  EAED,  dated  10  April  2007. 
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Use  of  OMB  Federal  Enterprise 

Architecture 

Reference  Models  (FEA  RMs) 
in  Support  of  NextGen  EA 


MITRE 


A  Proposing  a  new  FEA  BRM  Line  of  Business 

(LOB)  called  “Air  Transportation” 
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Proposed  Sub-functions 


Source:  Briefing  titled  “NextGen  Enterprise  Architecture  (EA)  T  I^E 

Focus:  Federal  Enterprise  Architecture  (FEA)  &  Federal 
Transition  Framework  (FTF)  Initiatives”  by  JPDO  EAED,  dated  10  April  2007. 


Additional  BRM  Revision  Proposals 


Proposing  an  additional  Sub-Function  under  existing  IT  Management  Line  of 
Business 


Existing  BRM  Lines  of 
Business 


Proposed  Sub-Function 


Proposed  update  to  current 
definition  to  include  facility  and 
equipment  certification 


Source:  Briefing  titled  “NextGen  Enterprise  Architecture  (EA) 

Focus:  Federal  Enterprise  Architecture  (FEA)  &  Federal 
Transition  Framework  (FTF)  Initiatives”  by  JPDO  EAED,  dated  10  April  2007. 


MITRE 
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Proposed  Extensions  to  Existing  SRM 


Proposing  a  new  “OperationalJBervices”  Domain  made  up  of  9  Service  Types 

and  26  Service  Components 


Proposed  Service  Components  MITRE 
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Using  the  Proposed  BRM  Reference  Model 

Extension 


j 


i 

i 


NextGen  EA 


Capobilfiy  102136 


Partner  Agency  EA 


Roadmap 


(Example  “FAA”) 


NASEA 


En  Route  Automation 

NextGen  Collision  Avoidance 

NextGen  Traffic  Flow 

SWIM 

Aeronautical  Data 

Modernization  (ERAM) 

System 

Management 

Link 

Source:  Briefing  titled  “NextGen  Enterprise  Architecture  (EA) 
Focus:  Federal  Enterprise  Architecture  (FEA)  &  Federal 


MITRE 


Transition  Framework  (FTF)  Initiatives” 


WSamSTOberations 


A  Use  of  FEA  Reference  Model  in  Support 

of  NextGen  EA  (cont’d) 


The  JPDO,  working  in  conjunction  with  the  OMB  FEA  office,  will  develop 
an  instantiation  of  the  five  OMB  FEA  reference  models  which  will: 

•  Form  the  core  of  the  enterprise  level  of  the  NextGen  EA.  The 
JPDO  will  apply  the  reference  models  to  support  comparisons 
of  architectures  from  participating  agencies  and  organizations. 

•  Help  identify  gaps  and  overlaps  in  capabilities,  operations, 
information  and  systems. 

•  Help  identify  key  areas  of  commonality  across  the 
organizations  that  will  require  improved  integration  and 
coordination. 

•  Frame  and  organize  the  NextGen  target  (to-be)  architecture  so 
that  participating  agencies  can  align  with  its  goals  and  content. 

•  Ensure  traceability  from  the  NextGen  target  architecture  to  the 
FEA  reference  models  to  facilitate  architecture  and  investment 
review. 
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Use  of  the  DoD  Architectural 
Framework  (DoDAF)  to 
Support  the  Development  of 
the  NextGen  EA 


MITRE 


Summary  of  NextGen  EA  DoDAF 

Products 


DoDAF  Product 

Purpose 

Overview  and  Summary  (AV-1) 

The  AV-1  gives  the  reader  a  condensed  overview  of  the  content  of  the 
architecture,  including  information  about  the  reason  tor  developing  the 
architecture,  the  architectural  products  that  will  he  developed  to  support  that 
purpose,  the  kinds  ot  analyses  to  be  conducted  on  the  products,  who  will 
conduct  them,  and  how  the  results  will  be  used,  The  AV-1  provides  a 
summary  ot  the  vision,  mission,  architecture  products,  assumptions  and 
constraints,  and  tools  used  in  the  process  ot  developing  the  architecture. 

Integrated  Dictionary  (AV-2) 

The  AV-2  is  the  NextGen  EA's  common  consistent  language,  reducing 
confusion  and  increasing  understanding  tor  partner  agencies  and 
stakeholders,  It  enables  the  JPDO  to  collaborate  and  socialize  the  NextGen 
EA  with  a  consistent  lexicon  formatted  in  such  a  way  that  stakeholders  can 
compare  the  elements  ot  the  NextGen  EA  to  their  respective  integrated 
dictionaries  and  architectures  and  propose  changes  it  necessary  to  ensure 
the  community  is  using  terms  consistently  and  as  intended. 

Community  Model  (OV-1) 

The  Community  Model  (OV-1)  is  used  to  provide  a  high  level  pictorial 
description  of  the  major  organizational  and  operational  elements  ot  the  Next 
Generation  Air  Transportation  System  and  how  those  elements  interact  with 
each  other  and  with  their  environment, 

Operational  Node  Connectivity 
Description  (OV-2) 

The  operational  nodes  shown  in  the  NextGen  EA  represent  idealized  places 
or  organizations  where  activities  are  performed  that  require  and/or  produce 
information.  The  OV-2  is  provided  not  only  to  represent  the  various 
organizations  and  places  where  activities  are  performed  but  also  to  provide 
a  framework  tor  organizing  the  information  exchanges  that  will  be  described 
in  detail  in  the  Operational  Information  Exchange  Matrix,  OV-3, 

Operational  Information  Exchange 
Matrix  (OV-3) 

The  OV-3  is  the  most  comprehensive  description  ot  operational  information 
flow.  It  further  decomposes  the  information  flows  (needlines)  identified  in  the 
OV-2  into  one  or  more  (usually  many)  specific  Information  Exchanges  and 
collects  the  detailed  information  about  each  exchange. 

Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED,  MITRE 
dated  22  June  2007. 
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Summary  of  NextGen  EA  DoDAF 

Products  (cont’d) 


DoDAF  Product  Purpose 


DoDAF  Product 

Purpose 

Activity  Model  -  Hierarchy  Node  Tree 
(OV-5) 

The  OV-5  Operational  Activity  Hierarchy  provides  a  comprehensive 
structured  list  ot  all  the  operational  activities  of  interest  to  the  architectural 
description.  That  is,  it  shows  all  the  activities  that  need  to  be  considered  to 
achieve  the  desired  outcomes  ot  the  architectural  analysis  and  how  high 
level  activities  are  composed  ot  lower  level  activities,  The  NextGen  EA  GV- 
5  Hierarchy  also  shows  how  the  operational  activities  align  within  the  eight 
Enterprise  Segments  and  the  relevant  Activity  Services  in  the  Community 
Model. 

Activity  Model— ID EFO  (Activity  Flow 
Diagrams)  (OV-5) 

The  OV-5  Activity  Flow  diagram  provides  information  useful  in  streamlining, 
combing,  or  eliminating  activities,  and  it  is  helpful  tor  eliminating  redundancy. 
Ot  the  greatest  importance  is  that  the  OV-5  Activity  Flow  diagram  provides 
the  foundation  tor  the  OV-3  and  OV-6c  diagrams  that  are  necessary  to 
understand  issues  ot  information  distribution  and  timing  -  critical  issues  tor 
developing  a  NextGen  EA  that  will  provide  the  performance  improvements 
required  by  the  NextGen  Vision. 

Operational  Eve  ntTT  race  Description 
(OV-6c) 

The  Operational  EvenbTrace  Description  provide  a  time  ordered  sequence 
ot  information  exchanges  among  the  operational  activities.  Because  the 

JPDO  will  use  the  architecture  as  a  basis  tor  advising  stakeholders  on  the 
value  and  criticality  ot  improving  their  systems  and  processes  to  achieve  the 
goals  of  NextGen,  an  understanding  of  the  sequencing  ot  operational 
activities  could  provide  kev  insiahts  in  suddou  otthat  Guidance. 

Operational  Activity  to  Stakeholder 
Investment  Traceability  Matrix  (SV- 
5  Hybrid) 

To  assist  Stakeholder  Communities  ot  Interest  in  aligning  the  NextGen  EA 
activities  to  their  respective  architectures  and  Capital  Investment  Plans, 
JPDO  has  deviated  from  the  standard  SV-5,  Operational  Activity  to  System 
Function  format  prescribed  by  DoDAF,  The  NextGen  SV-5  Hybrid  relates 
Operational  Activities  to  the  Stakeholder  Investments.  By  relating  NextGen 
EA  Operational  Activities  to  Stakeholder  Investments,  the  SV-5Hybrid 
describes  the  key  relationships  between  Stakeholder  Capital  Investment  and 
NextGen  Capabilities. 
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Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Enterprise  Segment  Activity  Inventory  (I 
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Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Enterprise  Segment  Activity  Inventory 
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Source:  “Enterprise  Architecture  V2.0”, 
produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Air  Navigation  Support  FEA  RM 

Alignment 


Figure  52:  Air  Navigation  Support  FEA  RM 


-Airspace  Security  Operations 
-  Capacity  M  artagem  ent 
-Flow  Contingency  Management 


The  Ail’  Navigation  Support  Segment  provides  the  short  and  long-term  planning  aspect  for  ensuring  the  safe,  secure, 
and  efficient  access  to  the  airspace.  As  depicted  in  figure  13,  this  segment  is  primarily  categorized  under  the  Air 
Transportation  Line  of  Business.  It  describes  the  activities  associated  with  the  prevention  and  countering  of  external 
attacks  on  aircraft  and  other  airborne  vehicles,  the  planning  and  design  of  the  airspace,  and  the  monitoring  of  current 
airspace  conditions  to  reallocate  if  necessary.  The  following  table  further  describes  the  Air  Navigation  Support 
Segment  and  identifies  how  each  activity  aligns  to  the  other  architectural  layers  of  the  NextGen  E A  Reference 
Architecture. 


Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED,  MITRE 
dated  22  June  2007. 
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Air  Navigation  Support  Activity  Model  (OV-5) 


Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED,  /VII  I  Kt 
dated  22  June  2007. 
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Use  of  Operational  Activity  to  Stakeholder 
Investment  Traceability  Matrix  (SV-5  Hybrid) 


Agencies'  Investments  to 

NextGen  Operational  Activities  (SV-5H) 


Gperaf  (Activities 


Stakeholders',  FX  Technology  type,  Budgeted  and  When  Mapping 


Source:  “Enterprise  Architecture  V2.0”,  produced  by  the  JPDO  EAED,  Ml  I  Kb 
dated  22  June  2007. 
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Use  of  the  OMB  Federal 
Transition  Framework  (FTF)  in 
Support  of  the  NextGen  EA 


FTF  As  Alignment  Mechanism 


The  FTF  is  a  catalog  of  government-wide  initiatives,  providing  agencies  with  an 
information  source  to  support  EA  planning  and  improve  the  efficiency  and  effectiveness 
of  investments  to  realize  service  improvements  and  cost  savings,  The  JPDO  envisions 
the  NextGen  ATS  FTF  catalog  item  as  a  mechanism  for  ensuring  that  the  relevant 
NextGen  segments  and  capabilities  of  the  NextGen  EA  are  incorporated  into  the  JPDO 
partner  agency’s  respective  target  architectures,  transition  plans,  and  capital  planning 
processes. 
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Source:  “Joint  Planning  and  Development  Office  -  Program  Management  Plan  VI  .0  for  the 
Enterprise  Architecture  and  Engineering  Division”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 


MITRE 


Benefits  of  Use  of  FTF  in  Support  of 

NextGen  EA  Effort 


•  Increased  partner  EA  agency  alignment  with  NextGen  EA 

•  Increased  sharing  and  reuse  of  common,  cross-agency  business 
processes,  service  components,  and  technology  standards 

•  Increased  agency  collaboration 

Additionally,  partner  agencies  can  use  the  FTF  as  follows: 

•  Obtain  consistent,  complete,  and  detailed  information  about 
NextGen  ATS  more  quickly  to  inform  EA  and  capital  planning 
efforts 

•  Make  more  informed  decisions  about  investments 

•  Improve  the  effectiveness  (i.e.,  performance)  and  efficiency  (i.e., 
cost  and  schedule)  of  investments. 
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Source:  “Joint  Planning  and  Development  Office  -  Program  Management  Plan  VI  .0  for  the 
Enterprise  Architecture  and  Engineering  Division”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Usage  of  FTF  in  Support  of  NextGen  E 


Publish  (September  07) 

-  FTF  catalog  published 
*  Identity  relevant  Cross 
Agency  Initiatives  that 
must  be  included  in 
Agency  E  A 


ASSESS  (Feb/March  08) 

*  Agency  submit  EA  self- 
assessments  to  0MB 
-  0MB  reviews  &  assesses  EA 
and  Transition  Strategy  for 
relevant  cross-agency 
initiatives 


*  Agency  FY10  budgets 
submitted  to  0MB 

*  0MB  verifies  to  ensure  EA 
Transition  Strategies  reflect 

relevant  cross-agency 
initiatives 


♦  FTF  Publication 

<u 


| 

< 


Incorporate  NextGen  ATS  into  Agency-level  EA 


Self-asses s  Agency  EA 
alignment  with  NextGen  ATS 


+  FTF  Assess 


zr 


TC 


m 


Align  Agency  budget  submission 
with  agency-level  EA  and  NextGen 
ATS 


3f 


Verily 


c 
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£ 

M 

Cl 
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Align  relevant  Agency  programs  and  investments  with  NextGen  ATS 


Source:  “Joint  Planning  and  Development  Office  -  Program  Management  Plan  VI. 0  for  the 
Enterprise  Architecture  and  Engineering  Division”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Usage  of  FTF  in  Support  of  NextGen  EA 


Scenario  1 

Incorporate  NextGen  ATS 
into  agency-level  EA 


Scenario  2 

Self-assess  EA 
alignment  with  NextGen 
ATS 


Scenario  3 

Alight  agency  budget 
submission  with  agency- 
level  EA  and  MextGen 
ATS 


Agency 

Steps 


*  Rewew  FTF  Catalog  to 
determine  ittk  "dspplcahilty  and 
sec of  the  WextGen  ATS  for 
your  agency. 

*  U pdate  EA  Program  Plan  lo 
incorporate  tasks  to  develop 
or  update  agenc-y  EA  y* *ork 
products. 

*  Update-  target  EA  te  reflect 
NejdGsn  ATS. 

-  Conduct  gap  aneJ^sIs  bdwwn 
current  and  targe?  architecture 
lo  identify  gape  In  currant 
imptementation  ofthe 
NetfGen  ATS. 

■  UpiMe  EA  Tradition  Strategy 
to  iffwperate  tasks,  adivfies, 
arid  milestone  to  close  gaps 
between  the  current  ^nd  target 
architecture 

■  Submit  reutfed  EA  work 
products  and  EA  Transition 
Strategy  io  0MB  annual 


Review  OMB  EA  Assessment 
Framework  and  insteuciionsto 
define  requirements  forme 
deveiopuHiri  anc  submission 
of  agency  EA  self-assessment 
materials. 

Refer  to  FTF  Catalog  to  cSeiine 
MeurtGen  ATS  requirements  for 
integration  with  agsncy  EA  and 
the  EA  Transition  Strategy. 
Conduct  self-assessment  of 
agency  EA  work  products,  EA 
Transition  Strategy  and 
implementation  resiis  relative 
to  NextGen  ATS  requirements 
ard  assessment  guidance. 
Prepare  s*tf-asse$$me*l  score 
and  s Emission  materials  for 
NextGen  ATS  and  Incorporate 
into  agency  EA  self 
assessment  package 
Sennit  agency  EA  self- 
assessment  package  to  OMB. 


Develop  IT  Investment 
portfolio  in  accordance  wth 
agency  policy  and  procedure?. 
Portfolio  should  reflect 
programs  and  projects  defined 
by  She  EA.  Transition  Strategy 
to  implement  agency  EA 
including  the  NaxtGen  ATS  at 
desciibed  in  the  FTF  Catalog. 
Select  protects  to  be  Included 
in  tie  IT  investment  portfolio 
In  accordance  with  agency 
policy  .and  procedures.  Select 
factors  show  Include 
aliyiment  with  agency  EA. 
Induing  the  NextGen  ATS. 
Submit  IT  investment  portfolio 
to  OMB  m  pari  ofthe  agency 
budget;  submission. 


EA  ass^smsnt  submission 
package. 


Scenario  4 

Align  agency  IT  programs 
with  NextGen  ATS 


*  Conduct  regular  control  and 

e  .aK*ate  actions  ae  part  erf  the 
CPIC  process  to  review  the 
alignment  of  current  programs 
with  agsncy  EA  and  EA 
Transition  Strategy  mending 
the  ItaitGen  ATS. 

*  Develop  recommendations 
and  corrective  actions  to  re- 
align  programs  wilh  agency 
guidance  and  Ihe  NewtGen 
ATS  as  described  in  the  FTF 
Catalog. 

*  Update  Individual  program 
plans  to  incotporat® 
recommendations  and 
corredive  actions. 

*  Update  IT  investment  portfolio 
to  reflect  recommendations 
and  ccwractive  actions. 
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Source:  “Joint  Planning  and  Development  Office  -  Program  Management  Plan  VI  .0  for  the 
Enterprise  Architecture  and  Engineering  Division”,  produced  by  the  JPDO  EAED, 
dated  22  June  2007. 
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Lessons  Learned 


MITRE 


Before  All  Else,  Articulate  the  Goals, 
Objectives  and  Purposes  of  the  EA  and 
Communicate  to  All  Stakeholders 


Know  why  you 
are  building 
the  architecture 


T 
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A  Articulated  and  Effective  Governance  is 

the  Foundation  of  the  EA  Effort 


*  Governance 

-  Defines  how  decisions  are  made,  who  has  the  authority  and 
who  is  accountable 

-  Determines  or  exerts  guiding  influence  over  program  policy, 
principles,  architecture,  strategy,  and  investment  and 
prioritization 

-  Determines  process  by  which  stakeholders  articulate  their 
interests  and  their  input  is  absorbed 

•  Governance  Framework 

-  The  plan  and  processes  that  describe  the  authority  structures, 
roles,  and  responsibilities  that  must  be  implemented  to 
effectively  govern 


An  EA  Program  without  articulated,  effective  governance  places 

the  entire  EA  program  at  significant  risk 


MITRE 
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Development  and  Use  of  a 
Communication  Strategy 


*  A  “How  to  Engage”  or  “Rules  of  Engagement” 
Guide  developed  for  use  by  external  stakeholders 
for  aligning  their  enterprise  life  cycle  processes 
and  EA  related  activities.  This  guidance  would  be 
a  communication  vehicle  that  would  articulate: 

-  The  enterprise  life  cycle  and  schedule  of  the  JPDO 
as  it  relates  to  the  development  and  release  of  the 
EA,  budget  and  R&D  guidance. 

-  The  pertinent  roles,  bodies  and  associated 
governance  structures  that  stakeholders  will  need 
to  be  cognizant  of  and  interact  with  on  an  on-going 
basis. 
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•  A  substantive,  explicitly  articulated  and  meaningful 
Operational  Concept  Graphic  (OV-1)  is  critical 

•  Issue  of  Utility  -  the  importance  of  the  Overview  and 
Summary  (AV-1) 

-  Significant  challenge  faced  by  architects  is  the  issue  of 
utility  of  the  EA 

•  Identifying  the  communities  who  need  the  EA 

•  Ensuring  the  info  is  in  the  proper  form  to  be  of  use 

•  Federation  DMZ 

-  Need  flexibility  to  manage  multiple  practices  and  levels 
of  maturity  i.e.,  need  a  DMZ 

•  Relating  the  EA  to  decision  making 

-  Coordinating  Federal  and  Industry  Business  Case 
Practices 

•  Closing  the  Design 

-  Issue  of  how  to  cope  with  long  planning  horizon  i.e., 
target  =  2025... 
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Summary 


MITRE 


— 


Summary 


*  NextGen  is  complex  system  with  many  public  and 
private  stakeholders  that  must  smoothly,  promptly, 
and  capably  integrate  with  the  changes  in  the  global 
air  transportation  system. 

*  EA  is  the  foundation  and  means  for  articulating  the 
vision  and  successfully  integrating  the  activities  of 
these  diverse  stakeholders  in  achieving  NextGen 


This  is  the  copyright  work  of  the  MITRE  Corporation  and  was  produced  for  the  US  Government  under  Contract  Number  DTDA01-01-C-00001  and  is  subject  to  Federal  Aviation  Administration  Acquisition 
Management  System  Clause  3.5-13,  Rights  in  Data  General,  Alt.  Ill  and  Alt.  IV  (Oct.,  1996).  *  m  .yn|- 

MITRE 
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Questions 


MITRE 


Backup 


AlCAASD 


MITRE 


_  An  Enterprise  Level  Focus  on 

NextGen 

Transformation  of  the  Nation’s  Air  Transportation 
System  requires: 

-  New  policies  to  create  the  right  relationships  and 
behaviors, 

-  Modernization  of  infrastructure  to  reduce  cost  and  set  the 
stage  for  a  new  level  of  performance,  and 

-  R&D  to  create  new  functionality  and  capability  that  takes 
advantage  of  a  modernized  infrastructure 

-  Engagement  of  multiple  entities  both  public  and  private 


Enterprise  Architecture  provides  the  holistic 
structure  to  manage  the  transformation  of  the 
National  Air  Transportation  System 


Source:  Briefing  titled  “The  Next  Generation  Air  Transportation  System  - 
EAED  Products  Status”  by  Dr.  Edgar  Waggoner,  dated  16  May  2007. 
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How  will  we  use  Enterprise 

Architecture? 


•  Enterprise  Architecture  as  a  documentation  method  provides  a  modeling 
framework  for  the  enterprise 


Documentation  of 
the  Future 
Enterprise 
Architecture  -  “to 

be” 


•  Enterprise  Architecture  enables  effective  management  of  NGATS  planning 
and  development  by: 

-  Supporting  change  management  at  the  various  levels 

-  Identifying  integrated  decisions  leading  to  requirements,  agency 
commitments,  and  ultimately  to  synchronized  investments  to  deliver  the 
NGATS 


-  Tracing  key  dependencies  between  capabilities,  policies,  operations, 
organizations,  systems,  and  technologies 


MITRE 
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Source:  Briefing  titled  “The  Next  Generation  Air  Transportation  System  - 
EAED  Products  Status”  by  Dr.  Edgar  Waggoner,  dated  16  May  2007. 


Use  of 


OMB  FEA  Reference  Models  in 
Support  of  NextGen  EA 
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Re-Forming  the  DoD 
Acquisition  Process 


A  Systems  Engineering 

Approach 

MR.  STEVE  WARD 
MR.  CHRIS  PERKINS 
DEPARTMENT  OF  THE  AIR  FORCE 
AERONAUTICAL  SYSTEMS  CENTER 
WRIGHT-PATTERSON  AFB,  OH 
22  OCT  2007 


DISCLAIMER 


THE  VIEWS  EXPRESSED  IN  THIS  PRESENTATION  ARE 
THE  VIEWS  OF  THE  AUTHORS  ONLY  AND  DO  NOT 
NECESSARILY  REFLECT  THE  VIEWS  OF  THE 
DEPARTMENT  OF  DEFENSE,  THE  DEPARTMENT  OF 
THE  AIR  FORCE,  AIR  FORCE  MATERIEL  COMMAND, 
OR  THE  AERONAUTICAL  SYSTEMS  CENTER. 


OVERVIEW 


•  CURRENT  DoD  5000  MODEL 

•  FAA  CERTIFICATION  PROCESS  MODEL 

•  PROPOSED  AIRCRAFT  ACQUISITION  MODEL 


Current  DoD  5000  Model 

All  Systems 


IOC 


FOC 


Concept 

Refinement 


Concept 

Decision 


Technology 

System  Development 

Production  & 

Operations 

Development 

&  Demonstration 

Deployment 

&  Support 

Design 

FRP 

Readiness  /\ 

/\  Decision 

Review  \/ 

\/  Review 

SRR  PDR  CDR  PRR 


Current  DoD  5000  Model 
All  Systems 

•  ADVANTAGES 

-  Framework  allows  flexibility 

-  Easily  tailored  for  specific  program  requirements 

-  Allows  for  Technology  Development  prior  to  SDD  phase 

•  DISADVANTAGES 

-  Most  risk  is  on  acquisition  agency  for  development 

-  Capability  and  certification  requirements  are  not  integrated 

-  Certifications  can  have  significant  impact  on  program  cost  and 
schedule 


Commercial  Development  Process 
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FAA  Certification  Process 


•  FAA  process  is  regulatory  -  Type  certification  requirements  are  must 
dos 

-  In  DOD  airworthiness  requirements  are  not  even  Key  Performance 
Parameters 

•  Customers  involved  in  creating  requirements  -  Notice  of  Proposed 
Rulemaking 

-  No  buy-in  by  customers  on  DoD  airworthiness  criteria 

•  Type  cert  board  establishes  criteria  up-front 

-  Includes  compliance  method 

-  Done  prior  to  design  and  test  phase 

•  Cost/schedule  of  compliance  is  better  known  up-front 

-  DoD  criteria  are  not  fully  agreed  to  until  after  cost  established 

•  Type  certification  drives  significant  cost  to  a  commercial  program 

-  AF  51 6B  drives  cost  but  those  costs  are  unknown  at  contract  award 

•  There  is  a  known  process  in  place  to  certify  components  -  Technical 
Standards  Orders  Database 

•  Independent  organization  verifies  compliance 


FAA  Certification 
Process 

•  ADVANTAGES 

-  Proven  safety  track  record 

-  Well  understood  cost  and  schedule 

-  Total  requirements  set  known  at  program  approval 

-  Early  planning  for  validation  minimizes  risk 

•  DISADVANTAGES 

-  Little  consideration  for  cost  of  ownership 

-  All  development  risk  is  on  the  airframe  developer 

•  Design  influenced  by  available,  mature  technology 


Proposed  Model 
Aircraft  Systems 
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Proposed  Model 
Aircraft  Systems 

•  Milestone  A  -  Technology  Development 

-  Entry  criteria 

•  Technology  Development  Strategy 

•  Initial  Capabilities  Document 

-  Contract  type  -  Cost  Plus/Award  Fee 

-  Timeline/Schedule 

-  Integrated  Risk  Assessment 


Proposed  Model 
Aircraft  Systems 

•  Milestone  B  -  Technology  Maturation  and 
System  Design 

-  Entry  criteria 

•  SRR 

•  Capabilities  Description  Document 

•  Certification  Plan 

-  Contract  type  -  Cost  Plus/Incentive  or  Award 
Fee 

-  Timeline/Schedule 

-  Integrated  Risk  Assessment 


Proposed  Model 
Aircraft  Systems 

•  Milestone  C  -  System  Integration  and 
Certification 

-  Entry  criteria 

•  CDR 

•  Capabilities  Production  Document 

-  Contract  type  -  Fixed  Price/Incentive  Fee 

-  Fixed  Timeline/Schedule 

-  Integrated  Risk  Assessment 


Proposed  Model 
Aircraft  Systems 

•  Milestone  D  -  Production 

-  Entry  criteria 

•  PRR 

•  System  Certification 

•  Successful  Initial  Operational  Test  &  Evaluation 

-  Contract  type  -  Fixed  Price/Incentive  Fee 

-  Timeline/Schedule 

-  Integrated  Risk  Assessment 


Proposed  Model 
Aircraft  Systems 


•  ADVANTAGES 

-  Integrates  systems  engineering  events  with  acquisition 
milestones 

-  Integrates  capability  and  certification  requirements 

-  Utilizes  a  known  development/certification  process 

-  Allows  risk-based  management  of  resources 

-  Provides  Time  Certain  certification  -  similar  to  FAA 

-  Similarity  to  FAA  cert  encourages  broader  business  base 

•  DISADVANTAGES 

-  Increases  the  number  of  Defense  Acquisition  Boards 


Summary 


•  Current  acquisition  process  has  room  for 
improvement 

•  Requirements  and  acquisition  processes  need  to  be 
better  integrated 

•  Program  risk  can  be  reduced  through  better 
alignment  of  acquisition  milestones  and  systems 
engineering  events 
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October  23,  2007 
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*  Foreword 

*  Introduction 

•  Purpose 

•  Vision 

•  Scope 

*  Objectives  (5) 

*  Actions  (40) 

■  Action 

■  Rationale  (why  it’s  needed) 

■  Discussion  (implementation  guidance) 

■  Lead  &  supporting  organizations 

■  Products  (what  is  expected) 

■  Completion  goal  (year) 

*  Execution  Management 


http://www.acq.osd.mil/ds/se/publications/AMSMP_041706_%20FINAL2.pdf 
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Five  Objectives,  40  Actions 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


1-1  M&S 

management 

1-2  Model-based 
systems 
engineering  & 
collaborative 
environments 

1-3  M&S  in  testing 

1-4  M&S  planning 
documentation 

1-5  RFP  &  contract 
language 

1-6  Security 
certification 


Key 

Broader  than  Acqn 
Partially  broader 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


2-1  Product 
development 
metamodel 

2-2  Commercial 
SE  standards 

2-3  Distributed 
simulation 
standards 

2-4  DoDAF  utility 

a)  DoDAF  2.0 
Systems 
Engineering 
Overlay 

b)  Standards  for 
depiction  & 
interchange 

2-5  Metadata 
template  for 
reusable 
resources 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


3-1  Acquisition 
inputs  to  DoD 
M&S  priorities 

3-2  Best  practices 
for  model/sim 
development 

3-3  Distributed  LVC 
environments 

a)  Standards 

b)  Sim/lab/range 
compliance 

c)  Event  services 

3-4  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

a)  Prioritize  needs 

b)  Pilot  projects 

c)  Expansion  as 
warranted 


Objective  4 


Improve 
model  and 
simulation 
use 


4-1  Help  defining 
M&S  strategy 

4-2  M&S  planning 
&  employment 
best  practices 

4-3  Foster  reuse 

a)  Business  model 

b)  Responsibilities 

c)  Resource 
discovery 

4-4  Info  availability 

a)  Scenarios 

b)  Systems 

c)  Threats 

d)  Environment 

4-5  W&A 

a)  Documentation 

b)  Risk-based 

c)  Examination 
4-6  COTS  SE  tools 

4-7  M&S  in  acqn 
metrics 


Objective  5 


Shape  the 
workforce 


5-1  Definition  of 
required  M&S 
competencies 

5-2  Harvesting  of 
commercial 
M&S  lessons 

5-3  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

5-4  M&S  education 
&  training 

a)  DAU,  DAG  & 
on-line  CLMs 

b)  Conferences, 
workshops  & 
assist  visits 

5-5  MSIAC  utility 


Objective  1 :  Provide  Necessary  Policy  &  Guidance 


1-1 .  Provide  effective,  persistent  DoD-wide  M&S  management  to  address 
cross-cutting  M&S  issues,  coordinate  actions 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E,  etc. 
Products:  Revised  DoDD  5000.59  (M&S  Management)  (&  DoDI)  with  clearer  responsibilities, 
revised  EXCIMS  (MSSC)  membership,  SOP  for  EXCIMS  (MSSC)  processes,  a 
refocused  DMSO  (MSCO) 

Completion  goal:  2006 


j  rfew  DoD  JVJ&3  structure  \n  pfece;  effectiveness  j/j  doubt 

•  No  acquisition  community  leadership  role  (Tri-chair)  on  MSSC 

j  rJew  DoD  Directive  oOOOfed  flmdly  released  Aljcj  D7,  but  defining  bay 
responsibilities  and  processes  a  wan  a  n  DoDI 

•  Current  project  selection  process  is  inconsistent,  inefficient,  and  wastes  $ 

•  Action  completion  is  overdue  (2006) 


Next  Steps: 

•  Continue  to  argue  for  an  SSE  leadership  role  on  M&S  SC 

•  Push  for  a  DoDI  on  M&S  management 

•  Propose  an  alternative  DoD  fe&S  pfenning  approach 


Objective  1 :  Provide  Necessary  Policy  &  Guidance 


1-2.  Promote  model-based  systems  engineering  (MBSE)  and  M&S-enabled 
collaborative  environments,  at  both  the  program  and  joint  capability  level 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  Components 
Products:  Revised  guidance  in  DAG 

Completion  goal:  2007 


•  Current  DAG  mentions  collaborative  environments  14  times, 
3J//JLjJHi:j£;rj-b2J3sd  faslincj  onoo.j  SBA  twice,  i\nd  DJDGD  noi  in  sjJL 

•  Programs/companies  often  claim  collaborative  environments,  bui  only  psinkii 

•  MBSID  sj  prominent  part  of  INCOSE’s  GE  Vision  2 020;  MBSE  Initiative 
underway 

•  increasing  industry  use  of  MBSIb  concept  &  tools 

•  AMSWG  (SSE)  submitted  new  DAG  language 


Next  steps: 

•  Nothing  further  now.  Reconsider  if  submitted  DAG  language  is  rejected. 
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Objective  1 :  Provide  Necessary  Policy  &  Guidance 


1-3.  Establish  policy  and  guidance  on  appropriate  use  of  M&S  to  plan  tests,  to 
complement  system  live  tests,  and  to  evaluate  joint  capabilities 

Co-leads:  OUSD(AT&L)/DS,  ODOT&E;  Support:  Components 
Products:  Revised  policy  and  guidance  in  DoDI  5000.2  and  DAG 

Completion  goal:  2007 


j  Coucepis  siccBpisd,  but  JiriJs  prsjuucsjJ  pujdsiuue  r^psjrdjjjp  cm^rki  for  M&S 
use 

j  JbJETD  Juuucbed  but  nmny  cbsjJJe/jpes  ubeud,  j/JcJudl/Jcj  poJjoy 

•  increased  discussion  of  M&S  support  to  testing  in  latest  submission  to  M&S 
section  of  DAG 

•  NDIA  DD&D  Cmte  js  coordinating  development  of  industry  recommendations 
fur  changes  to  ~f&E  portions  of  DoDD  5000  series  (&  possibly  DJDDJ  3170.01) 


Next  steps: 

•  NDIA  M&S  Cmte  participate  in  DT&E  Cmte  effort 

•  Track  JME  fC  polcy  development,  react  as  required 

•  Drafl i/submi  changes  to  T&E  portions  ©if  DoLDI  5000.2  &  [DAG 
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Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


1-4.  Establish  policy  to  require  documented  M&S  planning  at  the  joint 
capability  &  program  levels  as  part  of  the  Systems  Engineering  Plan, 
T&E  Strategy  and  T&E  Master  Plan 

Co-leads:  OUSD(AT&L)/DS(SSE),  ODOT&E;  Support:  Components 
Products:  Revised  policy  and  guidance  in  DoDI  5000.2,  DAG,  and  DOT&E  TEMP 
Planning  Guidance 

Completion  goal:  2007 


•  AMSWQ  (SSE)  submitted  revised  language  to  DoD  3000,2,  DAG  Janguacj 
and  SEP  Preparation  Guide 

j  PsirihiJ  siccspEjnce  of  SEP  Ejljs  hir,  oiher  TED 

j  j\lo  3i cIjdjtj  ibis  hir  r9£|3jrdj/j£j  Jancjuacja  j;j  TEEJE  EJ/jcj  Guidance 


Next  steps: 

•  Drafilsubmit  language  for  TEMP  Planning  Guidance 


Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


1-5.  Establish  M&S-related  guidelines  for  solicitations,  source  selections, 
and  contracting. 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OUSD(AT&L)/DPAP,  ODOT&E,  Components 
Products:  Sample  language  in  DoD  publications  (e.g.,  DAG,  SEP  Preparation  Guide, 
Contracting  for  Systems  Engineering  Guidebook)  regarding  M&S  requirements,  data 
rights,  and  the  responsibilities  and  liabilities  of  parties  regarding  sharing  and  reuse 

Completion  goal:  2007 


•  GoJjcjied  inputs  from  AMSWQ  members  and  industry  (through  NDIA  M&S 
Cmte) 

j  >\jVJ £> V V Gj  (SSE)  submitted  DAG  language  regarding  source-selection  criteria 
j  Presentation  at  Dei:  07  NDIA  Systems  Engineering  Conference 


Next  steps: 

•  further  refinement  and  vetting  of  proposed  guidance 

•  Synthesize  best  language  &  submit  to  DAG  (update),  SEP  Preparation  Guide, 
and  Contracting  for  Systems  Engineering  Guidebook 
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Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


1-6.  Ensure  practical  guidelines  for  information  assurance  certification 
and  accreditation  of  M&S  federated  networks  falling  under  multiple 
Designated  Accreditation  Authorities  (DAAs) 

Lead:  OASD(NII);  Support:  OUSD(AT&L)/DS(SSE),  OUSD(I),  NSA 
Products:  Proven,  practical  guidelines  published  in  DAG  and  DoD  8500. 2-H,  per 
DoDI  8500.2  “Information  Assurance  Implementation,”  Feb  6,  2003 

Completion  goal:  2007 


j  j\IJ]  bus  pubJjsbsd  DuD]  8500,2,  but  AMBWG)  rjussnons  nducjunuy 

•  AMSWG-NIJ  discussions  underway;  NIJ  reviewing  NAVAIR  procedures  fur 


suitability  jn  DAG 
UnJjkeJy  to  uuj/jpJefe  j n  2007 


Next  steps: 

•  Continue  to  ground  discussions  in  practical  experience;  push  Nil  as 
warranted 

•  Draft,  vet,  sind  submit  DAG  language 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 


2-1 .  Develop  a  product  development  information  metamodel  &  associated 
metadata  extensions  to  the  DoD  Discovery  Metadata  Specification 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OASD(NII),  Components 
Products:  Revised  DDMS;  revised  guidance  in  DAG. 

Completion  goal:  2008 


•  JSF  has  developed  si  metamodel  specification  and  provided  it  to  MSCO 

j  Ji  saarr is  unJjbaJy  MSCO  M&S  CO\  Dlscovary  bJarsidsiisi  pro jaoi:  wiJJ  siddraoo 
siriyiriinp  bay  3 /id  disco  vary  rnarsidsihi 

j  Jor  bopao  to  ariJisi  MSCO  (bcruddar)  sissisisinaa  to  avoJva  its  rrjatsijriodaJ 

Next  steps: 

•  Explore  iRA&E  interest  to  make  this  a  “blue”  effort 

•  Cooperate  with  JSF  in  efforts  to  revise/extend  metamodel 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 


2-2.  Support  development  of  open  commercial  and  non-proprietary 

standards  for  systems  engineering,  such  as  OMG’s  Systems  Modeling 
Language  (SysML)  and  ISO  Standard  10303  AP-233 

Co-leads:  OUSD(AT&L)/DS(SSE);  DoD  CIO  Support:  OASD(NII),  DLA, 
OUSD(AT&L),  Products:  Standards  suitable  for  use  by  DoD 

Completion  goal:  2007 


j  SyslML  vl.O  issued  us  an  “available  standard;”  v  1/J  minor  revision  late  2003 

•  Increasing  usage  &  teaching  of  SysML;  major  subject  at  JNCOSE,  NDIA 

•  Navy  M&S  Standards  Steering  Group  Jias  proposed  SysML  us  a  standard 

•  AP-233  SP  data  Interchange  standards  being  released  incrementally 
j  COTS  System  Lrjpjrjssrjrjp  too is  incorporating  SysML  and  AP-233 

j  id  Ahj/jp  yei  submiLsd  to  DoD  Sisifidsirdjsuusf]  Propru/ri 


Next  steps: 

Track  SysML  and  AP-233  implementations,  publicize  results 
•  investigate  DoD  Standardization  Program  process;  submit  SysML  and  AP-233 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 


2-3.  Establish  a  forum  to  clarify  the  characteristics  and  application  of 
various  distributed  simulation  standards  (ALSP,  DIS,  HLA,  SI3,  TENA, 
etc.)  and  examine  opportunities  for  convergence 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/TRMC  &  DS(SSE),  ODOT&E, 
Components 

Products:  (1 )  Information  on  strengths  &  weaknesses  of  the  various  standards;  (2) 
agreement  on  policy  and/or  guidance  on  the  use  of  distributed  simulation  standards; 
(3)  a  way  ahead  regarding  distributed  simulation  standards 

>  Completion  goal:  2007 


•  MSSC-funded  LVCAR  Project  underway,  but  bablnd  schaduJs 

•  SE  Forum  Is  interested,  had  taken  cm 9  briefing 

•  AfvJSWG  members  engaged  im  this  effort  and  tracking  progress;  ccrjccrrj  on 
rccjLJjrc/jJcrjis  dcbrjlbcrj 

Next  steps: 

•  No  idditiomilJ  steps  needed 
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Obj.  2:  Enhance  the  Technical  Framework  for  M&S  (cont.) 


2-4.  Improve  the  utility  of  the  DoD  Architecture  Framework  (DoDAF)  for 
acquisition 

2-4(a)  Develop  Systems  Engineering  Overlay  (pjKDfile)  for  DoDAF  v2.0 

Lead:  O^SD(AT&L)/DS^&upport:  OAS£)(NII),  Components  / 

Products:  Acquisition  JOverlay  for  DoDAF  v2.0  /  / 

Comjzrfetion  goal:  2006  /  /  / 

2-4(b)  Support  development  of  open  commercial  standards  for  the 
depiction  and  interchange  of  DoDAF-compliant  architectures 

Lead:  OASD(N II)  Support:  OUSD(AT&L)/DS(SSE) 

Products:  Published  standards  suitable  for  adoption  by  DoD  in  DoDAF  2.0;  revised 
guidance  in  DAG 

Completion  goal:  2007 


•  2-4(a):  Overlay  concept  has  been  dropped,  so  this  action  is  OBE 

•  2 -4(b):  OMG’s  UPDM  (UML  Profile  for  DoDAF/MODAF)  nearing  finalization 
°  SE  Forum  just  beginning  to  appreciate  the  value  of  DoDAE 

j  ASD(NII)  Is  attempting  to  make  DoDAF  v2.0  more  useful  for  acquisition 


Conmiuimy  psinJclpHilo/j  Yn  DoDAF  WGj  curDijJod 


Next  steps: 

•  Increase  acquisition  community  involvement  in  DoDAF  WG,  Including 
pushing  for  commercial  standards  for  architecture  data  exchange 

•  Revise  AMSMP  to  eliminate  Action  2-4  (a) 
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Obj.  2:  Enhance  the  Technical  Framework  for  M&S  (cont.) 


2-5.  Establish  a  standard  template  of  key  characteristics  (metadata)  to 
describe  reusable  M&S  resources 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE)  &  TRMC,  OASD(NII), 
ODOT&E,  Components 

Products:  Published  standard  template;  usage  guidance  in  DAG 

Completion  goal:  2007 


j  MSCO  M&S  COJ  Discovery  Metadata  project  underway  to  address  this 
•  Usrjcjs  guidance  in  DAQ  wiii  follow  downstream,  after  template  definition 


Next  steps: 

•  Support  MSCO  metadata  project  by  participating  in  reviews 

•  investigate  OMG’s  Reusable  Asset  Specification  (RAS) 
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Objective  3:  Improve  Model  &  Simulation  Capabilities 


3-1 .  Establish  a  process  to  ensure  acquisition  needs  are  reflected  in  DoD 
M&S  priorities 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E,  DOD  CIO, 
Components 

Products:  A  method  to  capture  and  prioritize  acquisition  needs. 

Completion  goal:  2007 


j  AMSWG  has  successfully  obtained  M&S  SC  funding  for  several  projec 

j  jVJCCC  fundj/jp  bus  pone  io  projeoio  or  sjueoborisjbJe  value,  perhaps  becsi uos 
AbJCWC  wsj s  loo  rnodesi:  1 n  wbai  Yt  proposed 

j  Aosjujolborj  JdDC  odd  does  noi:  have  sj/j  erfeobve  voice  In  orber  DoD  njndlnsj 
arenas  rbsii  owe ol  DJDC  capability,  sucb  sis  other  DDT  sir jd  D/\KP/\ 


Next  steps: 

•  investigate  DoD  S&T  planning  process  to  identify  entry  points 

•  Bui  id  list  of  acquisition  M&S  S&T  needs 
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Objective  3:  Improve  Model  &  Simulation  Capabilities 


3-2.  Define  and  foster  best  practices  for  efficient  development  and 
evolution  of  credible  M&S  tools,  incorporating  user-defined 
requirements,  a  systems  engineering  approach,  and  appropriate 
verification  &  validation 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E,  DOD  CIO, 
Components 

Products:  Best  practices  publication,  available  via  MSIAC,  DTIC,  etc.;  DAG  guidance 
to  use 

Completion  goal:  2008 


Mo  3j£jrjjfjcsim  obori  ibus  far 

Expool  ooino  j/jsjcjbis  iroin  olhor  bool  praaijaas  saJjabaijor.)  (Aobon  4-2) 


Next  steps: 

•  Conduct  Jiterature  search;  synthesize  fees t  practice 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (cont.) 


3-3.  Enable  readily-available  distributed  live-virtual-constructive 
environments,  leveraging  related  initiatives 

3-3(a)  Establish  DoD-wide  standards  for  distributed  environments 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/TRMC  &  DS(SSE);  ODOT&E;  DOD  CIO, 
Components 

Products:  Published  standard;  DODI  (#  TBD)  policy  to  use 

Completion  goal:  2008 

3-3(b)  Make  candidate  simulations,  labs  and  ranges  compliant  with 
these  standards 

Lead:  Components;  Support:  OUSD(AT&L)/DS(SSE)  &  TRMC,  ODOT&E 
Products:  A  larger  collection  of  simulations,  labs,  and  ranges  ready  to  be  employed  in 
distributed  events 

Completion  goal:  2010 


3-3(c)  Ensure  availability  of  services  to  help  plan  and  conduct  events 

Lead:  Components;  Support:  OUSD(AT&L),  OUSD(AT&L)/TRMC,  DISA 
Products:  Fee-based  technical  services  to  help  users  (e.g.,  PMs,  Capability  Managers, 
OTAs)  plan  and  conduct  distributed  events 

Completion  goal:  2009 


VC  Architecture  Roadmap  project  underway 


jMoibj/jp  becjun  o/j  other  ctendorde  (object  rnodeJCj  dote  ejscboncje,  etc.) 


Next  steps: 

•  Investigate  use  of  SysRyJL  in  Federation  Development  end  Execution 
Process  (FEDEP) 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (cont.) 


3-4.  Centrally  fund  and  manage  the  development  of  high-priority,  broadly- 
needed  M&S  tools 


3-4(a)  Identify  and  prioritize  broadly-needed  M&S  tools 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE);  ODOT&E,  DOD  CIO, 
Components 

Products:  Prioritized  list  of  common  M&S  tool  needs 

Completion  goal:  2007 

3-4(b)  Conduct  one  or  more  pilot  projects  to  develop  new  M&S  tools  or 
update  existing  ones  to  meet  these  needs  prove  this  mngmt  concept 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  Components 
Products:  Proof  of  concept  for  managing  the  development/evolution  of  M&S  tools  to 
meet  broadly-shared  needs 

Completion  goal:  2008 


3-4(c)  Expand  the  scope  of  central  M&S  tool  management  as  warranted 
by  pilot  project  results  and  the  list  of  common  M&S  needs 


Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E,  Components 
Products:  Capability  to  provide  broadly-needed  M&S  tools  in  a  more  responsive  and 
cost-effective  way. 

Completion  goal:  2011 


j 


AM  B  W  Gj  3  u bnj j  trod 
projoors  for  M  BBB 


3-4  (b)  pjJor  propoosiJ^  os 
f  Y03  rLJfjdJfjp 


orio  or  roo  rop  v  oopujsjrjorj 


Next  steps: 

•  Conduct  central  model  management  pilot  project 
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Objective  4:  Improve  Model  &  Simulation  Use 


4-1 .  Provide  potential  acquisition  M&S  users  the  knowledge  needed  to 
formulate  an  effective  M&S  strategy  via  ready  access  to  M&S  expertise 
and  information  about  M&S  capabilities  and  gaps,  reusable  resources, 
lessons-learned,  etc. 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE) 

Products:  Revised  guidance  in  DAG;  improved  knowledge  base  in  MSIAC;  assist  visits 
(e.g.,  by  OUSD(AT&L)/DS(SSE) 

Completion  goal:  2008 


*  j  i©V  J  >6 


d  guidance  submitted  to  DAG 

•  SSE  M&S  Cell  assisting  sis  able,  buz  n uz  widely  i\ dverdsed 

•  j\Io  older  Comporjejils  offer!  jtjcj  esslslenoe 

j  5-'J  EduosiiJor]  projeol  Idemlfled  in i\ny  JdDG  Bodies  of  Knowledge  (BoKs) 
Ibei  imnj  offer  useful  Imormsiljor] 


Next  steps: 

a  Advertise  and  expand  assist  visits 

•  Improve  MSIAC  expertise  regarding  M&S  in  acquisition  (Action  5-5) 
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Objective  4:  Improve  Model  &  Simulation  Use 


4-2.  Define  and  disseminate  best  practices  for  disciplined  M&S  planning  & 
employment 

Lead:  OUSD(AT&L)/DS(SSE),  Support:  OUSD(AT&L),  Components 
Product:  Revised  best  practices  guidance  in  DAG  and  MSIAC 

Completion  goal:  2007 


•  High-level  discussion  included  Itj  “M&S  for  Systems  Engineering”  CLM 

•  Expanded  discussion  submitted  in  recent  DAG  revision 

•  j  J&S  PJ21  jjjrjjrjcj  and  l^mploymeint  Best  Practices  solicitation  completed  in 
April 


Next  steps: 

•  Synthesize  best  practice,  conduct  AM  3  WG  &  j  JdJA  reviews 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-3.  Facilitate  the  sharing  of  reusable  resources 

4-3(a)  Establish  a  DoD-wide  business  model  for  compensating  providers 
of  reusable  M&S  resources  (e.g.,  information,  software,  services) 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Documented  business  model;  revised  policy  and/or  guidance  in  DoD  5000  series 
&  DAG 

Completion  goal:  2007 


MSSC-funded  M 


,v. 


Re 


source  Reuse  Business  Model  project  underway 


Next  steps: 

•  No  further  action  needed  yet 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-3.  Facilitate  the  sharing  of  reusable  resources 


4-3(b)  Establish  DoD  policy  and/or  guidance  regarding  responsibilities 

to  share,  protect  and  properly  use  M&S  information,  tools,  and  data 
Co-Leads:  OASD(NII),  OUSD(AT&L),  USD(I);  Support:  OUSD(AT&L)/DS(SSE)  & 
DPAP,  OUSD(P&R),  OUSD(C)/PA&E,  Components 
Product:  Revised  policy  and/or  guidance  in  various  issuances  (e.g.,  DoD  5000  series, 
DAG,  contracting  guidance) 

Completion  goal:  2008 


•  Drafted  and  submitted  DAG  language 

•  M&S  Resource  Reuse  Business  Model  projaci  may  make  recommendations 
on  this  subject 


Next  steps: 

•  Draft  language  for  contracting  guide 

•  (Hold-off  submitting  a  5000  change) 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 

4-3.  Facilitate  the  sharing  of  reusable  resources 


4-3(c)  Enhance  the  means  (e.g.,  directory  service,  registries,  bulletin 
boards)  to  discover  the  existence  of  reusable  resources  required  for 
M&S  and  contact  information 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  A  better  way  to  discover  reusable  resources.  Re-orientation  and  integration  of 
various  DoD  M&S  resources  repositories. 

Completion  goal:  2007 


j  A\  Sbaf/ar  has  dlraciad  MBBO  io  davaJap  a  “raJjabJa  rapaadary":;  wa  bav- 
abjaaiad  baa  ad  on  prararjLJjsnaa;  prajaai:  js  pracaadlrjg 

j  Aaba/ja  2-6  la  a  prarapLJJsda  ia  4-3  (a) 


Next  steps: 

•  Monitor  jJSCO  project;  no  further  notion  fj aadad  now 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-4.  Define  the  types  of  information  DoD  organizations  shall  make  available  to 
others  with  a  clearance  and  valid  need  to  know  and  the  processes  to  obtain 
them  (per  reuse  business  model).  The  process  to  obtain  information  should 
include  an  efficient  mechanism  for  industry  to  request  government  data  with 
specific  "need  to  know"  outside  a  specific  contract  environment. 

4-4(a)  Scenario  data 

Lead:  OUSD(AT&L)  Support:  OCJCS(J8),  OUSD(C)/PA&E,  DIA,  Components 
Product:  Approved  scenarios  and  process  to  obtain 

Completion  goal:  2007 

4-4(b)  System-related  data 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  ODOT&E,  Components 
Product:  Process  to  obtain  authoritative  system  data  (characteristics  and  performance, 
interactions,  interfaces,  logistic  support,  etc.)  documented  in  the  DAG  and  appropriate 
OASD  (Nil)  policy  documents. 

Completion  goal:  2008 

4-4(c)  Threat  data 

Lead:  DIA;  Support:  OUSD(AT&L);  OUSD(AT&L)/DS(SSE),  ODOT&E,  and 
Components 

Product:  Authoritative  threat  data  and  process  to  obtain 

Completion  goal:  2007 

4-4(d)  Natural  environment  data 

Lead:  DoD  Natural  Environment  MSEAs;  Support:  OUSD(AT&L), 
OUSD(AT&L)/DS(SSE),  Components 

Product:  Authoritative  natural  environment  data  and  process  to  obtain 

Completion  goal:  2007 
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Action  4-4  Assessment 


Acquisition  Support  Division  olf  DIA  has  briefed  AMSWG  and  NDIA  M&S  Cmte 
on  its  support  to  acquisition  programs 

jJJSJC  has  briefed  NDIA  jvI&S  Cmte  or j  YidAP  program  and  provided 
instructions  on  how  to  request  T  j  J AP  models 

Draft  DAB  language  discusses  threat  data  sources  and  traceabiiJity 

No  method  exists  “for  industry  to  request  government  data  with  specific 
‘need  to  know’  outside  a  specific  contract  environment” 

MSSC-funded  Environmental  Scenario  Generator  projaui  underway 

j\Jo  prupr^ss  rspurdinp  sburinp  LLS,  oyotorn  dsitu 

■Joint  Rapid  Suamj/iu  Geuamuuu  (JRSG)  and  Joint  Data  -AitiruatlyaG  (J DA) 

protests  adyarttes  tbay  wjJJ  address  aii  the  Rotten  4-4  into  needs;  time  wjJJ  teii 


Next  steps: 

•  Participate  in  JRSG  arte  JDA  effort  as  resources  permit 

•  investigate  data  sharing  polices  of  OSD,  JCS,  and  ©fiber  Components 

•  investigate  JSC,  PAIS,  &  Service  scenario  data  availability  &  access 

•  Draft  add'l  DAG  language  on  all  data  types  (interim  prior  to  JRSG  UDA) 

•  Use  NDIA  M&S  Cmte  to  assess  seriousness  of  no-contract  data  access  issue 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-5.  Foster  cost-effective  VV&A 

4-5(a)  Require  DoD-wide  standardized  documentation  of  VV&A 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E, 
Components 

Products:  Revised  policy  in  DODI  5000.2  and  5000.61;  revised  guidance  in 
DAG 

Completion  goal:  2007 


•  jvISCO  project  underway 

•  AMSWG  concern  that  draft  MSSC’s  “BoD  M&S  Strategic  Vision”  call  for 
“practical  verification,  validation,  and  accreditation  guidelines  that  vary  by 
application  area”  (emphasis  added)  will  undermine  W&A. 

Next  steps: 

•  Monitor  MSCO  project 

•  Actively  participate  in  siny  DoDI  5000.61  update 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-5.  Foster  cost-effective  VV&A 


4-5(b)  Develop  risk-based  methodology  and  associated  guidelines  for 
VV&A  expenditures 

T  Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  Components 

Products:  Updated  DoDI  5000.61;  revised  policy  and  guidance  in  DoDI  5000.2 
and  DAG 

Completion  goal:  2007 


•  jvJSCO  projeci  underway 

Next  steps: 

•  Monitor  MSCO  progress  developing  risk-based  W&A  guidelines,  take 
action  as  necessary 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-5.  Foster  cost-effective  VV&A 


4-5(c)  Examine  a  program’s  VV&A  when  M&S  informs  major  acquisition 
decisions  and  unambiguously  state  the  purpose,  key  assumptions  and 
significant  limitations  of  each  model/simulation  when  results  are  presented. 

Lead:  OUSD(AT&L)/DS(SSE)  Support:  DoD  Components 

Products:  Guidance  &  training  for  oversight  personnel;  updates  to  DAG  Chaps  4,  9 

Completion  goal:  2007 


•  Submitted  DAG  Janguage  on  W&A  examination 

•  SSD  M&S  CeJJ  has  given  initiall  briefing  to  OUSD(A&T)/SSE/AS 
j  jxJd  oibar  DD/upD/jarji  sjcijvlilas  u/jdsrwsjy 


Next  steps: 

•  Broaden  teaching  on  W&A  examination 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-6.  Assess  the  use  of  COTS  systems  engineering  tools  (modeling 
environments)  for  collaborative  architecture  development 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OASD(NII),  Components 
Products:  Revised  guidance  in  DAG;  enhanced  M&S  body  of  knowledge  for 
dissemination 
Completion  goal:  2007 


•  SysMIL  3j/j  d  AT -2 33  already  proving  utility 

•  UPDId  nearimg  finalization,  can  help  with  CADM  AMD  DAHS  weaknesses 

•  NIST  “Systems  Engineering  Tool  Interoperability  Plug-fest”  underway 

Next  steps: 

•  Jnvesti uate  use  off  SE  tools  for  coBfaborative  architecture  development 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-7.  Define  and  capture  meaningful  metrics  for  M&S  utility  in  acquisition 
Co-Leads:  OUSD(AT&L),  Dept,  of  the  Navy  Support:  OUSD(AT&L)/DS(SSE), 
Components 

Products:  Metric  definitions  in  DAG;  methods  to  capture  and  submit  data  in  DAG; 
data  from  individual  projects  in  MSIAC,  Body  of  Knowledge,  etc. 

Completion  goal:  2007 


Ojtjb  D'/'tha  iop  v  2JC£j LJJsjiJDjrj  MiiiB  projects  lor  JVJS3C  f  YDS  fu/jdjrjp 


Next  steps: 

•  Monitor  MSSC  project  if  funded 
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Objective  5:  Shape  the  Workforce 


5-1 .  Define  required  M&S  competencies  for  the  acquisition  workforce 

Co-Leads:  DAU  and  OUSD(AT&L)/DS(SSE);  Support:  OUSD(P&R), 
OUSD(AT&L)/DDRE,  OUSD(C)/PA&E,  Components 
Product:  Identified  lead  FIPT;  workforce  qualification  requirements;  management 
process  &  structure 

Completion  goal:  2008 


*  “Educating  the  M&S  Workforce”  project  underway  with  Navy  &  MSSC  funding 

j  r  YDS  fu/J djjrjp  bicre/riem  bej/ip  cojickJered  by  MBBC 


Next  steps:  No  further  activities  needed  now 
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Objective  5:  Shape  the  Workforce 


5-2.  Harvest  lessons  from  commercial  sector  activities  in  the  use  of  M&S  to 
support  product  development 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OUSD(AT&L),  Components 
Products:  Annual  update  to  best  practices  in  DAG  and  lessons  from  industry  that  should 
be  considered  by  PMs  in  planning  for  M&S 
Completion  goal:  Recurring;  initial  in  2007 


participating  irj  conferences,  workshops,  and  literature  review  involving 
commerciai  industry  use  of  jJ&S,  csipiLjrl rjcj  relevant  points 

•  3rjcr92igj/j£j  industry  adoption  olf  “Simulation-leased  Dssjcjtj 


Next  steps: 

•  CoBBect  and  consoIJidate  findings,  feed  into  Action  5-3  BoK 
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Objective  5:  Shape  the  Workforce 


5-3.  Assemble  and  evolve  the  M&S  Body  of  Knowledge  (information  set) 
relevant  to  acquisition 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  Components 

Product:  Information  base  available  to  potential  M&S  users  (e.g.,  PMs,  CMs,  OTAs); 

source  material  for  education  and  training 
Completion  goal:  Recurring;  initial  in  2006 


•  Several  BolKs  have  been  discovered 

•  Knowledge  jc  being  developed  (e.rj.j  beet  practices) 


Next  steps: 

^  Synthesize  a  consolidated  BoK,  put  Mo  configuration  management,  make 
accessible  (bow  much  of  this  is  accomplished  by  Education  Project  is  TBD) 
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Obj.  5:  Shape  the  Workforce  (cont.) 


5-4.  Educate  and  train  the  workforce  to  achieve  required  M&S 
competencies 

5-4(a»  Provide  M&S  knowledge  via  an  expanded  set  of  DAU  courses, 
the  Defense  Acquisition  Guide,  and  on-line  CLMs 

Lead:  DAU;  Support:  OUSD(AT&L),  OUSD(AT&L)/DS(SSE),  Components 
Product:  Expanded  set  of  DAU  courses,  improved  M&S  guidance  in  the  Defense 
Acquisition  Guide,  on  line  Continuous  Learning  Modules;  a  better  educated 
workforce 

Completion  goal:  2009 


•  CILM  jJ&S  for  BE  with 

•  CLM  on  M&S  forT&E 


over  1,600  graduates 
just  released 


DAG  wjJJ 


i 

D: 


uiburjced  us  jrjfDmjHiJD/j  denned 


Do  ecuon  on  DAU  courses  thus  fur 


Next  steps:  Nothing  further  needed  now;  BoK  is  prerequisite 
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Obj.  5:  Shape  the  Workforce  (cont.) 


5-4.  Educate  and  train  the  workforce  to  achieve  required  M&S 
competencies 


5-4(b)  Provide  M&S  knowledge  via  conferences,  workshops,  and 
assist  visits 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OUSD(AT&L),  DAU,  Components 
Product:  Annual  outreach  program;  a  better  educated  and  trained  workforce 
Completion  goal:  Recurring;  initial  in  2006 


•  f  YD/  Outreach  IPJan  approved  by  AMSWG;  includes  M&S  tutorial  for  AS 
staff,  DI/ISC,  NDIA  and  SISO  presentations 

j  Addd  jr/jHisrlsiJs  bssi  prsicilcss)  in  work 

•  Resource  constrained 


Next  steps: 

*  Advertise  and  expand  assist  visits 

•  Hold  workshops  once  recommended  practices  are  in  hand 
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Obj.  5:  Shape  the  Workforce  (cont.) 


5-5.  Improve  the  knowledge  and  expertise  available  through  the  MSIAC  to 

make  it  of  greater  utility  to  the  acquisition  community 
Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Plan  of  action  with  coordinated  MSIAC  CONOPS  &  staffing  requirement;  list  of 
knowledge  shortfalls  that  MSIAC  will  take  on;  success  criteria  &  process  to  bring 
MSIAC  up  to  criteria 

Completion  goal:  2008 


Dn\ y  corjversHijorjs  wjih  jVJSJAC  comrsicior  -'thus  h\r 


Next  steps: 

*  Develop  a  pk\n  of  action  to  improve  the  M&S  Information  Analysis  Center’s 
usefulness  to  the  acquisition  community 
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Background 


•  Currently  PhD  Systems  Engineering  (SE) 
Candidate  At  Southern  Methodist  University 

-  Presentation  Is  Part  Of  Dissertation  Research 


•  Worked  20+  Years  As  Systems  Engineer 

-  Supporting  Aircraft  Flight  Control  Systems 
Both  Commercial  And  Military 

-  SE  Consultant/Tool  Vendor 

•  Currently  Perform  Software  Release  Activities 
For  Military  Aircraft  Systems 


Introduction 


SMU 

- 1 - 

•  System  of  Systems  (SoS) 

-  Definition  From  INCOSE  Handbook: 

■  System  of  systems  applies  to  a  system-of-interest 
whose  system  elements  are  themselves  systems; 
typically  these  entail  large  scale  inter-disciplinary 
problems  with  multiple,  heterogeneous,  distributed 
systems. 

•  Definition  Contains  “Large  Scale  Inter-Disciplinary 
Problems”  For  A  Reason 

-  SoS  Breeds  Complexity,  Increasing  The  Integration 
Required  For  Software,  And  Increasing  The 
Complexity  Of  The  Decision  Process  For  When  To 
Release  Interim  Software 


Complexity  Of  SoS 


•  Working  On  Your  Own  Car  -  One  Of 
America’s  Past  Times 

-  Due  To  Complexity  Of  Cars  Today  - 
Almost  A  Thing  Of  The  Past 

-  Cars  Today  Contain  Greater  Complexity 
Than  Previous  Models 

■  A  Car  May  Contain  50  Microprocessors 

-  Future  Technologies  Driving  Additional 
Complexity 

■  Drive  By  Wire 


Complexity  Of  SoS  (Continued)  ^ 

•  Aircraft  Have  Always  Been  Complex  Systems 

•  Today,  Aircraft  Are  Incorporating  More  And  More 
Features 

-  Increases  Complexity  And  Integration  Of 
Systems 

•  Previously,  Flight  Control  Software  Could  Be 
Released  Without  System  Integration  Testing 
With  Avionics  Software 

-  Today  The  Software  Is  So  Tightly  Integrated 
Providing  Advanced  Automodes,  Even  The 
Slightest  Of  Changes  Requires  System 
Integration  Testing 


Software  Complexity ]X 

•  Joint  Strike  Fighter  (JSF)  Software  Growing 

-  General  Accounting  Office  (GAO)  Reported  To 
Congressional  Committees 

■  March  2006  -  Program  Would  Develop  19  Million 
Lines  Of  Code  (LOC) 

■  March  2007  -  Program  Would  Develop  22  Million  LOC 

-  In  One  Year,  Estimate  Increased  By  3  Million 
LOC  Or  16% 

-  Just  Think  Of  The  Complexity  Added  In  Those 
Previously  Unaccounted  For  3  Million  LOC 

-  Number  Of  Actual  Software  Releases  Not  Given 

■  Software  Delivered  Over  5  Blocks 

■  Assume  Software  Releases  Number  More  Than  The  5 
Delivery  Blocks 


Software  Complexity  (Continued)^ 

•  One  Word  - 

-  Windows  3.1 

■  Approximately  3  Million  LOC 

-  Windows  95 

■  Approximately  15  Million  LOC 

-  Windows  Vista  (Latest  Operating  System) 

■  Approximately  50  Million  LOC 

-  As  Everyone  Knows,  The  More  Code  Added, 
The  Less  Complex  Microsoft’s  Operating 
System  Became  -  Right? 

-  Multiple  Releases  For  Each  Operating  System 
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Software  Release 

•  Production  Software  Release 

-  Definition  As  Used  Here: 

■  A  Release  To  The  End  Customer  That  Is 
Validated  And  Verified  And  Meets  All 
Requirements 

■  End  Customer  Is  Customer  That  Receives 
The  Software  After  All  Verification  And 
Validation  Activities  Are  Complete 

-  Some  May  Call  This  Final  Release 

■  Requirements,  Design,  Verifications, 
Validations,  And  Testing  All  Complete 

-  Sense  Of  Accomplishment 


Software  Release  (Continued)  «£ 

•  Interim  Software  Release 

-  Definition  As  Used  Here: 

■  A  Release  That  Is  Not  Fully  Verified  Or 
Validated  To  All  The  Requirements 

-  Software  Release  Occurring  Before 
Production  Release 

-  Often  Times  Requirements,  Design, 
Verifications,  Validations,  And/Or  Testing 
Is  Not  Complete 

■  Interim  Releases  Used  To  Complete 
Design,  Verifications,  etc. 
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Software  Release  (Continued) 


SMU 

— I — 

•  Imagine  A  SoS  Program  Today 

-  Difficult,  If  Not  Impossible,  To  Proceed  With 
Only  A  Production  Software  Release 

-  The  Complexity  And  Integrated  Nature  Of  SoS 
Almost  Requires  Interim  Releases 

•  With  That  Said 

-  Take  Today’s  Integrated  SoS  Environment 

-  Add  In  Program  Schedule  Pressures 

-  All  Equals  Complex  Decisionology  Regarding 
When  To  Release  Interim  Software 

•  A  Process  Methodology  For  Assisting  In 
Interim  Software  Release  Decision  Making 
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Integrated  Process  Methodology  ^ 

•  An  Integrated  Process  Methodology  Being 
Developed  For  Assisting  Decision  Making 
Regarding  When  To  Release  Interim  Software 

•  Proposed  Integrated  Process  Methodology 
Considers: 

-  Incomplete  State  Of  Program  That  May  Exist  For 
An  Interim  Release 

-  Interfaces,  Problem  Reports,  Resources, 
Requirements,  Software  Criticality,  Schedules, 
etc. 

•  Assist  In  Determining  The  Optimal  Time  To 
Produce  An  Interim  Software  Release  Given 
Multiple  Release  Paths  And  Multiple  Integrated 
Software  Products,  While  Considering  The 
Factors  Mentioned  Above 
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Integrated  Process  Methodology  (Contjsyu 

•  Proposed  Integrated  Process  Methodology 
Not  Meant  To  Replace  Software  Planning 

-  Aid  In  The  Software  Release  Decision  Making 
Process 

-  Software  Plan  Used  As  An  Input  To  The 
Integrated  Process  Methodology  Decision 
Matrix 

•  Nor  Is  The  Process  Methodology  Meant  To 
Solve  The  Question  Of  When  To  Release 
Software 

-  Allows  The  Decision  Makers  To  Make  Better 
Decisions  Regarding  When  To  Release 
Software 
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Integrated  Process  Methodology  (Contjsyu 

•  Goals  Of  Integrated  Process 
Methodology 

-  Aid  In  Decision  Making  Process  By 
Provide  An  Analytical  Methodology 

-  Replace  Non-productive  Decision 
Methodologies  Currently  In  Use 

■  For  Instance  -  BOGSAT  (Bunch  Of  Guys 
Sitting  Around  Table),  Squeaky  Wheel,  etc. 

-  Replace  Multiple  Smaller  Software 
Releases  With  Fewer,  More  Integrated 
Releases 
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I ntegrated  Process  Methodology  (Cont)*™ 

•  Cost  And  Schedule  Normally  Constrain 
The  Number  Of  Software  Releases  For  A 
Particular  Program 

-  Normally,  Releasing  Software  Incurs 
Both  A  Schedule  And  Monetary  Cost 

■  Takes  A  Finite  Amount  Of  Time  To  Make, 
Build,  Release,  Document,  And  Minimally 
Test  Release 

■  During  The  Release,  Resources  Used 
(People,  Computers,  Labs,  Etc.)  Not 
Available  To  Perform  Other  Tasks 

-  Consequently,  Fewer  Software  Releases 
Equates  To  Less  Cost  To  Program 
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Future  Work 


•  Define  Generic  Interim  Software 
Release  Process 

-  Accounting  For  A  Program’s 
Unfinished  Nature  During  Interim 
Software  Releases 

•  Develop  And  Verify  The  Integrated 
Process  Methodology 
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Abstract 


This  paper  covers  factors  that  make  production  software  releases 
successful  -  design  complete,  requirements  complete,  testing  complete, 
customer  expectations  set,  etc.,  and  how  a  production  software  release  process 
may  not  be  fitting  for  an  interim  software  release  due  to  the  state  of  the  program 
during  an  interim  release  -  one  or  all  of  the  production  release  success  factors 
possibly  being  incomplete,  open  system  problem  reports,  interactions  with  other 
systems  -  hardware,  communications  and  software  interfaces,  schedule 
interactions,  resource  constraints,  etc.  A  discussion  will  follow  on  the  need  for  an 
integrated  process  methodology  that  factors  the  incomplete  nature  of  a  program 
during  an  interim  software  release  into  the  release  decision  methodology.  The 
goals  for  the  integrated  process  methodology  will  be  discussed  along  with  the 
next  steps  in  developing  the  integrated  process  methodology  for  interim  software 
releases. 

Introduction 

Today’s  complex  systems  are  becoming  more  and  more  integrated,  as 
evidence  by  the  growing  field  of  Systems  of  Systems  (SoS).  Consequently, 
software  is  being  integrated  with  other  processors  within  its  own  system  and 
across  interfaces  within  the  total  system  itself,  increasing  the  complexity  and 
integration  required  for  software  releases. 

SoS  Adds  Complexity 

SoS,  as  the  name  implies,  is  a  system  comprised  of  other  systems. 
Creating  a  system  composed  of  other  systems  adds  additional  complexity  and 
integration  challenges.  For  instance,  cars  today  may  have  50  microprocessors 
controlling  everything  from  the  engine  to  the  air  bag  [1].  Every  microprocessor 
runs  its  own  software  and  probably  interfaces  with  additional  microprocessors, 
driving  additional  complexity  and  integration  pains.  The  Drive  By  Wire  [2] 
technology  for  future  cars,  will  only  increase  the  complexity  and  integration 
challenges.  In  the  past  cars  could  be  serviced  by  mechanically  inclined 
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individuals  who  did  not  mind  getting  their  hands  dirty.  Today,  one  practically 
needs  a  degree  in  software  engineering  to  service  cars. 

Aircraft  have  always  been  complex,  integrated  systems,  but  today,  as 
more  systems  integrate  with  each  other,  the  aircraft  is  becoming  more  complex 
and  tightly  integrated.  Not  long  ago,  the  flight  control  software  (commands 
surfaces  to  keep  the  aircraft  flying)  could  be  released  with  minimal  to  no 
integration  testing  with  avionic  software  (controls  the  mission).  Today  the  flight 
control  and  avionic  software  are  tightly  integrated  providing  advanced  functions. 
This  integration  demands  that  even  the  smallest  of  software  changes  drives 
system  and  integrated  testing  to  insure  the  software  changes  did  not 
detrimentally  affect  the  aircraft  in  some  unforeseen  manner. 

The  complexity  and  integration  requirements  of  a  SoS  affects  the  system’s 
software  and  its  safety  implications.  As  Leveson  [3]  points  out: 

Today  we  are  building  systems  -  and  using  computers  to  control  them  - 
that  have  the  potential  for  large-scale  destruction  of  life  and  the 
environment:  Even  a  single  accident  may  be  disastrous. 

Today’s  added  complexity,  additional  requirements,  and  criticality  of 
software,  means  the  decision  of  when  to  release  software  is  becoming  as 
complex  as  the  software  itself.  This  paper  will  explore  a  software  release 
methodology  that  considers  the  complexity,  integrational  aspects,  and  criticality 
of  today’s  software. 

Software  Complexity 

Software  is  complex  and  becoming  more  complex  daily.  As  an  example, 
take  the  Joint  Strike  Fighter  (JSF)  program.  In  March  of  2006  in  a  report  from  the 
General  Accounting  Office  (GAO)  to  Congressional  Committees,  it  was  reported 
that  the  JSF  program  would  develop  19  million  lines  of  code  [4].  In  March  2007, 
the  GAO  reported  the  program  would  develop  22  million  lines  of  code  [5].  In  one 
year  the  estimate  increased  by  3  million  lines  of  code  or  16%.  Just  think  of  the 
complexity  added  in  those  previously  unaccounted  for  3  million  lines  of  code. 

The  JSF  software  was  to  be  delivered  in  5  different  blocks,  but  the  number  of 
actual  software  releases  was  not  given.  It  can  be  assumed  that  the  software 
releases  will  number  more  than  the  delivery  blocks. 

Looking  at  the  number  of  lines  of  code  can  give  an  idea  of  software 
complexity,  the  more  lines  of  code,  the  more  complex  the  software.  Looking  at 
Microsoft’s  LOC  count  shows  an  interesting  trend  [6]: 

Real  systems  show  no  signs  of  becoming  less  complex.  In  fact,  they  are 
becoming  more  complex  faster  and  faster.  Microsoft  Windows  is  a  poster 
child  for  this  trend  to  complexity.  Windows  3.1,  released  in  1992,  had  3 
million  lines  of  code;  Windows  95  has  15  million  and  Windows  98  has  18 
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million.  The  original  Windows  NT  (also  1992)  had  4  million  lines  of  code; 

NT  4.0  (1996)  has  16.5  million.  In  1998,  Windows  NT  5.0  was  estimated  to 
have  20  million  lines  of  code;  by  the  time  it  was  renamed  Windows  2000 
(in  1999)  it  had  between  35  million  and  60  million  lines  of  code,  depending 
on  who  you  believe. 

Windows  Vista,  Microsoft’s  latest  operating  system,  reportedly  contains 
50  million  lines  of  code  [7].  The  number  of  software  releases  for  a  product 
with  50  million  lines  of  code  has  to  be  large.  Imagine  performing  only  one 
software  release  for  a  product  with  50  million  lines  of  code. 

Software  Releases 

Given  today’s  integrated  environment,  releasing  production  software  is  an 
accomplishment  in  itself.  With  a  production  release,  the  design  is  complete, 
testing  is  complete,  requirements  are  verified,  outstanding  problems  are 
mitigated,  contractual  obligations  have  been  met,  the  schedule  no  longer  is  a 
plan,  it  is  the  actuals  for  the  program,  and  significant  management  oversight  - 
sometimes  known  as  help  -  is  provided,  making  the  path  for  a  production  release 
familiar  and  the  process  well  known.  Accompanying  a  production  release  is  a 
sense  of  accomplishment  for  a  job  well  done  and  possibly  the  end  of  the  program. 

With  all  that  said,  defining  production  release,  as  used  in  this  paper,  is 
required.  A  literature  search  will  discover  many  terms  and  definitions  related  to 
software  releases  [8,  9,  10,  1 1].  For  the  purpose  of  this  paper,  production 
software  release  will  be  defined  as  a  release  to  the  end  customer  that  is 
validated  and  verified  to  meet  all  the  requirements.  Along  the  same  lines,  an 
interim  software  release  is  a  release  that  is  not  fully  verified  or  validated  to  all  the 
requirements.  Customer,  as  used  here,  is  defined  as  a  user  of  the  software.  A 
customer  could  be  internal  or  external  to  the  company.  An  end  customer  is  the 
customer  that  receives  the  software  after  all  verification  and  validation  activities 
are  complete. 

In  today’s  integrated,  SoS  environment,  it  would  be  difficult,  if  not 
impossible,  to  proceed  through  a  production  software  program  of  any  size  with 
only  a  production  software  release.  The  complexity  and  integrated  nature  of  SoS 
almost  requires  interim  releases  before  the  production  release. 

If  the  path  to  production  release  is  well  known  and  familiar,  does  it 
necessarily  follow  that  the  production  software  release  path/process  is  adequate 
for  interim  software  releases?  Production  software  releases  benefit  from  the 
completeness  of  the  design,  testing,  requirements  and  problem  mitigations, 
interim  releases  usually  do  not  have  those  luxuries.  An  interim  release  usually 
contains  partial  functionality  and  may  even  occur  before  the  design  is  complete 
and  may  be  used  to  complete  requirement  verification  meaning  requirements 
may  not  be  verified.  Because  design  may  be  on-going,  testing  may  not  be 
complete,  requirements  may  still  require  verification,  and  outstanding  high 
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severity  problems  may  not  be  mitigated,  a  production  release  process  may  not 
suffice  for  an  interim  release.  Today’s  integrated  SoS  environment  along  with 
program  schedule  pressures  add  to  the  complexities  of  interim  release  decision 
making. 

Integrated  Process  Methodology 

An  integrated  process  methodology  is  being  developed  for  assisting  in  the 
decision  making  regarding  when  to  release  interim  software.  The  integrated 
process  methodology  will  consider  the  incomplete  state  of  the  program  that 
exists  for  an  interim  release  and  additional  factors  that  could  affect  a  release 
such  as  interfaces,  problem  reports,  resources,  requirements,  software  criticality, 
and  schedules.  The  integrated  process  methodology  will  assist  system 
development  programs  in  determining  the  optimal  time  to  produce  an  interim 
software  release  that  supports  its  intended  purpose,  given  multiple  release  paths 
and  multiple  integrated  software  products,  while  considering  the  factors 
mentioned  above. 

The  proposed  integrated  process  methodology  is  not  meant  to  replace 
software  planning,  but  aid  in  the  software  release  decision  process.  The 
software  plan  would  be  used  as  an  input  to  the  integrated  process  methodology 
decision  matrix  to  assist  in  determining  the  optimal  release  path  for  a  specific 
interim  software  release.  Nor  is  the  process  methodology  meant  to  solve  the 
question  of  when  to  release  the  software,  but  to  allow  the  decision  makers  to 
make  better  decisions  regarding  when  to  release  software.  The  methodology’s 
benefits  will  be  especially  useful  as  the  decision  of  when  to  release  software 
becomes  more  difficult.  Keeney’s  [9]  take  on  difficult  decisions  and  analysis: 

More  Difficult  decision  problems  are  naturally  more  difficult  to  analyze. 

This  is  true  regardless  of  the  degree  to  which  formal  analysis  (i.e.,  use  of 

models  as  a  decision  aid)  or  intuitive  appraisal  (i.e.,  in  one’s  head)  is  used. 

However,  as  complexity  increases,  the  efficacy  of  the  intuitive  appraisal 

decreases  more  rapidly  than  formal  analysis. 

Software  release  decisions  are  difficult  by  themselves,  but  when  combined 
with  the  problems  SoS  introduces,  there  may  be  too  much  information  required  to 
properly  process  the  decision.  The  decision  maker  may  then  use  simplified 
mental  strategies,  without  using  decision  analysis  methods  [10].  The  integrated 
process  methodology  would  be  used  to  analyzed  the  information  provided  and 
aide  in  the  decision  making  process,  with  the  goal  of  replacing  non-productive 
decision  methodologies  currently  in  use,  like  BOGSAT  (Bunch  Of  Guys  Sitting 
Around  Table).  Ideally,  the  integrated  process  methodology  will  provide  an 
analytical  methodology  to  aid  in  the  software  release  decision  process  with  the 
hopes  of  replacing  multiple  smaller  software  releases  with  fewer,  more  integrated 
releases. 
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The  goal  of  the  process  methodology  is  to  reduce  software  releases. 
That’s  a  good  thing,  right?  If  it  is  just  software,  can’t  it  be  released  anytime? 
While  it  is  true  that  software  can  be  released  anytime,  cost  and  schedule 
normally  constrain  the  number  of  software  releases  for  a  particular  program. 
Normally,  releasing  software  incurs  both  a  schedule  and  monetary  cost.  It  takes 
a  finite  amount  of  time  to  make,  build,  release,  document,  and  minimally  test  the 
release.  During  the  release,  the  resources  used  (people,  computers,  labs,  etc.) 
are  not  available  to  perform  other  tasks  (incurring  schedule  costs)  and  must  be 
paid  for  their  time  (incurring  monetary  cost).  Consequently,  the  fewer  software 
releases  needed,  the  less  the  cost  to  the  program. 

Future  work  includes  defining  a  generic  interim  software  release  process, 
developing  the  integrated  process  methodology,  verifying  the  process’  decision 
matrix,  verifying  the  integrated  process  methodology,  and  optimizing  the  process 
methodology. 
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•Waterfall 

•  Spiral 

*  Inc  re  menial 

Library  of 
Process-Related 
Documentation 

*  Project  Data 
•Corporate  Data 

•  Cost 

•  Sample 
Documents 

*  Templates 

•Sample 

Documents 

•Searchable 

Database 
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sFmm 

V 

Systems  Center 

Charleston 


EPO  website  provides  access  to  all 
of  SSC-C’s  organizational  process  assets 


Approximately  100  pages  of  content;  over  1000  documents  available 


SPAWAR 

V 

Systems  Center 
Charleston 


SSC-Charfeston  Engineering  Process  Office 


]  Search 


EPO  Home  |  ePlan  Builder  |  WBT  Courses  |  eWBS  |  Contact  EPO  |  CorpWeb 


Navigation 

Getting  Started 

EPO  Home 

Upcoming  Events 

Calendar 

Welcome  to  the  SPAWAR  System  Center  -  Charleston's  Engineering  Process  Office  (EPO)  Homepage.  This 

10/15/2OO7 
Airchitectinq  with 

SSC-C  Standard 

site  is  the  repository  for  a  wealth  of  systems  engineering,  software  engineering,  and  process 

DODAF 

Processes 

improvement  information  to  aid  our  vision  in  becoming  a  world-class  systems  engineering  organization. 

10/22/2007 

Process  Areas 

The  site  contains  the  SSC-Charleston  Organizational  Process  Assets,  including  the  organization's  set  of 

10th  Annual 

Systems 

Projects 

standard  engineering  processes  and  procedures,  tools,  sample  documents,  templates,  and  project 

Enqineerinq 

Conference 

Process 

guidelines.  The  measurement  repository  of  project  and  process  measures  is  also  accessible 

11/12/2007 

Improvement 

The  site  also  contains  information  about  the  Capability  Maturity  Model  for  Integration  (CMMI®)  and  3SC- 

7th  Annual  CMMI 

Teams 

Technology 

Conference  and 

Charleston's  commitment  to  process  improvement.  The  CMMI®  is  used  to  benchmark  and  measure  our 

Organizational 

process  improvement  progress  against  industry  best  practices. 

User  CrouD 

Measurement 

Repository 

Background 

more  » 

Training 

Innovation  Program 

SSC-C  is  committed  to  process  improvement  and  has  been  actively  pursuing  process  improvement 
since  1993.  SSC-C  is  implementing  the  Capability  Maturity  Model  for  Integration  (CMMI®).  The  IDEAL® 

Latest  Additions 

model  is  being  used  to  implement  process  improvement. 

2003  Innovation 

References 

Proqram 

•  SSC-C's  commitment  to  process  improvement  and  policy  regarding  it  were  re-affirmed  in  a  SSC- 

Application  &. 

Comments 

C  command-wide  Process  Improvement  Policy  dated  11  December  2003. 

Guidelines  NEW 

Please  direct  comments 

•  Navy  Endorses  CMMI  as  the  Standard  Process  Improvement  Model 

CMM#  Maturity 

about  or  problems  with 

*  ASN  RDA  Software  Process  Improvement  Initiative 

Level  4  Traininq 

this  site  to  the  EPO 

Brief 

Webmaster. 

The  information  below  describes  what  will  be  found  under  each  major  section  of  the  site. 

March  2QQ? 

H 
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Process  Area  Pages 


SHMR 

"r 

Systems  Center 

Charleston 


sptwm 


f 


]  Search 


SSC-Charleston  Engineering  Process  Office 


Systems  Center 
C/fa  Wes  ton 


Navigation 

Getting  Started 

Calendar 

SSC-C  Standard 
Processes 

Process  Areas 

Project  Planning 

(pp) 


EPO  Home  |  ePlan  Builder  |  WBT  Courses  |  eWBS  |  Contact  EPO  |  CorpWeb 


Project  Monitoring  &  Control  (PMC) 

Project  Monitoring  and  Control  (PMC)  is  a  Level  2  (Managed)  Process  Area.  The  purpose  of  PMC  is  to 
provide  an  understanding  of  the  project's  progress  so  that  appropriate  corrective  actions  can  be  taken 
when  the  project's  performance  deviates  significantly  from  the  plan. 

Policy  Document 

•  SSC-C  Project  Monitoring  and  Control  Policy 


Related  Process 
Areas 

Prelect  Planning 

fPP) 

Measurement  & 

Analysis  fMA) 


Project  Monitoring 
&  Control  (PMC) 

Configuration 
Management  (CM) 

Process  and 
Product  Quality 
Assurance  (PPQA) 

Requirements 

Management 

(REQM) 

Measurement  Et 
Analysis  (MA) 

Supplier  Agreement 
Management  (SAM) 

Requirements 
Development  (RD) 

Technical  Solution 

(T S) 


Process  Manual 

*  SSC-C  Project  Monitoring  and  Control  Process  Manual 


SOPs 

•  In  Process  Review  SOP 

•  Project  Management  Review  SOP 

•  Meeting  SOP 

Sample  Documents 

•  IBFTC  PMC  Plan 

•  CICS  Project  Management  Plan  (PMP) 

•  Towed  Array  Earned  Value  Plan 

Templates 

•  PMP  Plan 


Each  CMMI  process  area 
has  a  standard  page  with 
links  to  policy,  process 
manual,  SOPs, 
Sample/Project  documents, 
and  other  resources 


T I 

— ih  SCI  Par 
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Projects  Section 


SHMR 

"r 

Systems  Center 

Charleston 


SPAWAR 

V 

Systems  Center 
Charleston 


SSC-Charleston  Engineering  Process  Office 


j  Search 


ERO  Home  |  ePlan  Builder  |  WBT  Courses  |  eWBS  |  Contact  EPO  |  CorpWeb 


Navigation 


Getting  Started 

Calendar 

SSC-C  Standard 
Processes 

Process  Areas 

Projects 

SCAMPI  Appraised 

Projects 

Self  Assessed 
Projects 


Process 

Improvement 

Teams 

Organizational 

Measurement 

Repository 


Standard  CMMI  Appraisal  Method  for  Process  Improvement 
(SCAMPI)  Appraised  Projects 

SSC-C  SCAMPI  Appraisal  Summary 

SPAWAR  Systems  Center  -  Charleston  -  Maturity  Level  3 

*  Sponsor:  Mike  Kutch 

*  Projects  Appraised:  AP,  CICS,  IB  FTC,  JTWS,  SCN,  VIDS,  NAVMACS  II,  SSES,  Towed  Array 

*  Appraised  27  April  2007 

Integrated  Battle  Force  Training  Center  (IBFTC)  -  Maturity  Level  3 

*  Program  Manager:  Lexine  Langley 

*  Code  856 

*  Appraised  26  January  2007 

Visual  Information  Display  System  (VIDS)  -  Maturity  Level  3 

*  Program  Manager:  Steve  Whitbeck 

*  Code  663 

*  Appraised  15  December  2006 


Each  appraised  project 
has  a  page  and  is 
expected  to  share  good 
examples  of  plans  and 
documents 


Training 

Innovation  Program 
References 


Naval  Modular  Automated  Communications  System  n  (NAVMACS  II)  -  Maturity  Level  3 

*  Program  Manager:  John  Dyar 

*  Code  523 

*  Appraised  15  December  2006 


Comments 


Shipbuilding  and  Conversion,  Navy  (SCN)  -  Maturity  Level  3 


-i 


Tec 
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SRWAR 

▼ 

Systems  Center 

Charleston  _ 

Tools 


>  ePIan  Builder 

>  Organizational  Measurement  Repository 

>  Appraisal  Wizard 
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ePIan  Builder  Tool 


SP/MR 


V 

Systems  Center 
Charleston 


Space  and 
Wjir  Naval  Warfare 

Systems  Center 
Charleston 

ePIan  Builder 

Electronic  CM  MI1-1  Compliant  Documentation  Application 

Save  Quit  Help 

Spansor&d  by  efts  Dimct&f  of  Qp&fattobs  { QQK)  -  Micha&l  Kutch 

ePIan  Builder  tool 

-  An  interactive,  web-based  application  that  leads  the  user  through 
a  structured  interview  process  (like  TurboTax®)  to  generate  a 
CMMI®-compliant  plan 

-  Includes  standard,  consistent  text 

-  Generates  an  initial  project-specific  document 

•  Project  Management  Plan  (with  Work  Breakdown  Structure) 

•  Configuration  Management  Plan 

•  Process  and  Product  Quality  Assurance  Plan 

•  Requirements  Management  Plan 

•  Measurement  and  Analysis  Plan 

•  Supplier  Agreement  Management  Plan  (by  end  of  2007) 

•  Systems  Engineering  Plan  (DoD  SEP  Format)  TIechsoft" 

Technical  Software  Services,  Inc. 
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EPB  -  Select  Tasks  for  each  Role 


SP/MR 

y 


Systems  Center 
Charleston 


SfftWAR ,  Spaca  and 

_  «  Naval  Warfare 

Systems  Center 
Charleston 

ePIan  Builder 

Electronic  CMMI®  Compliant  Documentation  Application 

JP"  ->• 

,4 

r 

¥ 

Save  Quit  Help 

sponsored  by  £/?a  Dinchor  of  EnpiriG&rlflg  Op&ratiaris  (09K)  -  httctiabl  Kt/lci I 

cl  Setup 


Docurrienl 


Build 


PMP 


Tailor  each  role 
from  pre-defined 
list  of  tasks  and/or 
add  custom  tasks 


ORGANIZATION 

Organization 

Organization 

Chart 

P  ro  gram 
Manager  Tasks: 

Project  Leader 
Tasks 

Systems 

Engineering 

Tasks 

Security 
Engineering 
Ta  s  k  s 

Software 

Engineering 

Tasks 

Test 

Engineering 

Tasks 

0  o  nf  i  g  u  rati  o  n 
Manager  Tasks 

Quality 


Project  Leader  Tasks 

The  Project  Leader  is  responsible  for  establishing  and  maintaining  the  project  plan. 

Please  identify  the  specific  responsibilities  of  the  Project  Leader. 


P 

P 

P 

P 

P 

P 

P 

P 

P 


Coordinates  all  activities  of  the  prime  contractor  and  subcontractors 
Assigns  specific  responsibilities  to  subcontractors  [PP  GP  2.4] 

Discusses  technical  issues  from  the  Government  with  subcontractors 
Discusses  technical  issues  from  the  subcontractors  with  the  Government 
Manages  the  project  cost  and  schedule  [PMC  1.1]  ◄ - 


Resolves  any  inconsistencies  in  the  reguirements  [PMC  2.2] 
Mitigates  project  risks  [PMC  1.3] 

Manage  and  resolve  corrective  actions  [PMC  2.2]  [PMC  2.3] 

Provides  prime  contractor  and  subcontractor  work  products  and 
deliverables  to  the  Government 


Note  mapping 
to  CMMI® 
generic  and 
specific 
practices 


Please  enter  any  additional  specific  responsibilities  of  the  Project  Leader. 
Task 
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Work  Breakdown  Structure  (WBS) 
in  a  Project  Management  Plan 
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Risk  Identification  in  PMP 


sFmm 

T 

Systems  Center 
Charleston 

Risks 


This  page  allows  you  to  enter  a  list  of  known  or  expected  risks.  The  severity  of  the  risks 
and  the  mitigation  approach  for  each  should  be  identified.  Please  use  the  table  below  to 
identify  the  major  risks  associated  with  the  project. 

^  Click  for  more  information  about  risks 


Risk  Category 

|  Schedule  | 

Risk  Category 

I  Quality  ~r~| 


Risk  Category 


Technics 


Impact/Concern 

Level 

Mitigation  Approach 

Products  are  required  by 
the  customer  by  10/1/06 

1 

|  |  High 

J 

Be  prepared  to  provide  draft 
materials  if  development  of 

1 

Impact/Concern 

Level 

Mitigation  Approach 

Will  products  be  ready  for 

3 

|  Medium 

J 

Provide  technical  data  to  contractor 

3 

10/15/06  in  a  condition 

3 

1 

in  accordance  with  schedule  with 

3 

Impact/Concern 

Level 

Mitigation  Approach 

Ability  to  getteh 
technical  ata  from  the 

3 

1  |  High 

J 

Interact  directly  with  the  satellite 
manufacturer  to  obtain  the  technical 

1 

ALXj  iVlUltf  I  Ullli 


PMP  may  also  reference  a  more  comprehensive  Risk  Management  Plan 


Te 
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SP/MR 

T#- 


Measurement  &  Analysis  Plan 


Systems  Center 
Charleston 


Cost, 

Schedule,  and 
Process 
Performance 
are  standard 
categories  of 
measures 


Collection, 
Storage,  and 
Analysis  is 
defined  for 
each  Project 
measure 


Cost  is  a  measure  within  the  Financial  Performance  category  that  measures  the  cost  for 
activities,  events,  and  products.  The  measure  provides  an  easy-to  understand  view  of 
the  budget.  Comparison  of  planned  and  actual  cost  data  provides  insight  into  significant 
and  repetitive  cost  changes  at  the  activity  level. 

While  more  detailed  cost  information  provides  more  insight  into  the  project's  total  cost, 
until  the  project  personnel  have  achieved  a  certain  level  of  proficiency  in  estimating 
costs,  it  is  recommended  that  the  cost  data  should  be  captured  at  a  level  commensurate 
with  this  level  of  experience. 

Collection  and  Storage 

Identify  the  level  of  detail  for  capturing  cost  data 
|  Project  Level 

Please  select  how  the  Project  Leader  will  report  contract  costs  from  the  list  below.  If  the 
Project  Leader  is  not  responsible  for  managing  contracts,  select  "Project". 

|  Project  Z] 

Identify  who  will  provide  the  actual  cost  data: 

[Project  Leader 


Identify  the  tool  to  be  used  to  collect  cost  data: 

|BSA  and  PMACS 

Identify  how  often  the  actual  cost  data  will  be  collected: 
|  Monthly  13 

Analysis  Procedures 

Identify  how  often  the  cost  data  will  be  analyzed: 

|  Monthly  ~~U 

Identify  the  cost  alert  threshold: 

I  95%  -  | 


SEIRarts'ier 
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Systems  Engineering  Plan  (SEP) 


SRWNl 

"r 

Systems  Center 
Charleston 


SEP  format  follows  the  DoD  SEP  Preparation  Guide 


SEP 


NAS  Pensacola 
OSP  Survey 


gf  PROJECT  SETUP 

DOCUMENT 

SETUP 

gf  PROGRAM 

$  INTRODUCTION 

rf  ACQUISITION 
HISTORY 


Previous  Life- 
Cycle  Phases 

Next  Life-Cycle 
Phase 


rf  SYSTEM 

CAPABILITIES 

SE 

ORGANIZATIONAL 

INTEGRATION 

SYSTEM 

ENGINEERING 

PROCESS 

$  INTEGRATION 

INTEGRATED 
MASTER  PLAN 


D  50  100 

Next  Life-Cycle  Phase 

The  SEP  requires  that  the  program's  acquisition  history  and  life-cycle  phase  be 

discussed,  describing  the  top-level,  technical  process  used  in  each  life-cycle  phase,  This 
Next  Life-Cycle  Phase  section  should  give  an  overview  of  the  next  planned  life-cycle 
phase  as  well  as  summarize  the  process  activities  that  are  expected  to  be  finished  during 
the  next  life-cycle  phase, 

^lease  enter  text  discussing  the  Next  Life-Cycle  Phase  of  the  program. 

This  description  should  give  an  overview  of  the  planned  SE  process  and  should  have 
more  detail  than  the  historical  life-cycle  processes  completed,  It  should  include  how  the 
technical  process  will  be  integrated  into  the  life-cycle  model  and  summarize  the  process 
activities  that  are  expected  to  be  finished  during  the  next  life-cycle  phase, 

Life-Cycle  Phases  (in  hierarchical  order): 


1, 

2. 

3. 

4. 

5. 


Concept  Refinement 
Technology  Development 
System  Development  and  Demonstration 
Production  and  Deployment 
Operations  and  Support 


Show  Hidden  Text 
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Systems  Engineering  Plan  (SEP) 


SP/MR 


Systems  Center 
Charleston 


SEP 

NAS  Pensacola 
OSP  Survey 

PROJECT  SETUP 

DOCUMENT 

SETUP 

PROGRAM 

INTRODUCTION 

ACQUISITION 

HISTORY 

SYSTEM 

CAPABILITIES 

System 

Capabilities 

Certification 
Requirements 

Design 

Considerations 


SE 

ORGANIZATIONAL 

INTEGRATION 

SYSTEM 

ENGINEERING 

PROCESS 

INTEGRATION 


5^ 


SBC 


Design  Considerations 

This  section  describes  any  design  considerations  that  must  be  integrated  into  the 
engineering  design  effort  including  any  special  constraints  that  must  be  considered. 

Please  enter  any  design  constraints. 


These  design  constraints  are  any  special  considerations  that  must  be  taken  into 
account  before  they  are  integrated  into  the  project  during  the  engineering  process.  The 
text  should  also  describe  the  basis  for  these  design  constraints  and  how  the  technical 
authority  is  going  to  be  engaged  in  considering  and  integrating  these  constraints. 

Some  examples  of  design  constraints  are  as  follows: 

.  The  system  shall  be  able  to  operate  using  the  three  phase  power  available  on 
board  a  ship, 

.  The  system  shall  be  able  to  fit  into  a  standard  19"  rack. 

While  these  constraints  look  like  requirements,  they  are  not  system  requirements 
because  they  do  not  specify  what  the  system  must  do,  nor  do  they  specify  how  well 
the  system  must  perform  a  capability;  they  constraint  the  possible  solutions  by  limiting 
the  choices  available  to  the  engineers,  and  are  therefore  design  requirements  that 
constrain  the  solution  space. 


The  nature  of  the  SEP  requires  more  open  input  text  fields,  but 
EPB  helps  by  providing  elaborations  and  examples  for  the  user 

— S El  Partner 
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SEP  -  Planned  Trade  Studies 


SHMR 

Systems  Center 

Charleston 


SEP 

NAS  Pensacola 
OSP  Survey 

gf  PROJECT  SETUP 

rf  DOCUMENT 
SETUP 

^  PROGRAM 

INTRODUCTION 

□  ACQUISITION 
HISTORY 

rf  SYSTEM 

CAPABILITIES 

SE 

ORGANIZATIONAL 

INTEGRATION 

rf  SYSTEM 

ENGINEERING 

PROCESS 

Planning 

Process 

Improvement 

Modeling  and 
Simulation 

Resources 
Trade  Studies 


INTEGRATION 
^  INTEGRATED 


3  50  100 

Trade  Studies 

This  section  should  include  a  brief  description  of  the  process  used  to  determine  trade-offs 
between  various  attributes  of  the  program  (e.g.j  between  requirements  and  design), 
Information  about  how  trade  studies  are  addressed  within  the  organization  will  be 
automatically  embedded  into  the  document,  To  view  the  embedded  information  about  how 
trade  studies  will  be  addressed;  click  the  "Click  to  view  the  embedded  trade  studies  text" 
link  below. 

Q  Click  to  view  the  embedded  trade  studies  text. 

Trade  studies  will  be  addressed  in  accordance  with  the  SSC-C  Technical  Solutions 
Process  Manual  and  SSC-C  Decision  Analysis  and  Resolution  Process  Manual  where  the 
development  of  alternate  solutions;  selection  criteria  and  trade  processes  are  discussed. 


The  actual  trade  studies  to  be  performed  on  the  program  will  be  captured  and  listed  in  the 
control  below, 

Please  enter  the  trade  studies  that  will  be  conducted  on  this  program, 

Trade  Study 

Research  on  OSP  topologies 


Trade  Study 

Research  on  different  conduit  installation 


Te 
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SP/MR 

T#- 


ePB  Output 
SEP  Table  of  Contents 


Systems  Center 
Charleston 


Table  of  Contents 


SPMAR 

w£ 

N65236-393-PMP-000  1 

Ai^rt  13,2006 

Project  Management  Plan  (PMP) 

For 

MARSOC  Tflfest  SCAMPI  CER  (593) 

August  13,2006 

Prquxod  hy 

Sp  ue  and  Naval  Warfare  Syrians  Carter,  Chadaten 
(SSC  C) 

P  0  Box  190022 

North  Chads  ten,  SC  29419  5542 

Antraraito:  Mai:  Eami  (393 )  Auaust23.2006 

i 

1 .  Introduction . 

1.1  Pro  gam  D  escription  and  Applicable  D  o  cuments . 

1.2  T  echnical  Status  as  of  the  date  of  this  SEP . 

1 . 3  Approach  o  f  SEP  Up  dates . 

2.  System  Engineering  Application  to  Life- Cycle  Phases . 

2.1  Ac  quisition  History . 

2.1.1  Previous  Life- Cycle  Phas es . 

2.1.2  Next  Life- Cycle  Phase . 

2.2  SystemCapabilities,  Requirements  and  D  esign  C  onsiderations . 

2.2. 1  System  Capabilities . 

2.2.2  Certification  Requirements . 

2.2.3  DesignC onsiderations . 

2.3  SE  Organizational  Integration . 

2. 3. 1  Organizational  Roles . 

2.3.2  Pro gam RolesandResponsibilities . 

2.4  Training . 

2.5  System  Engineering  Process . 

2.5.1  Planning . 

2.5.2  Process  Improvement . 

2.5.3  Mo  deling  and  Simulation . 

2.5.4  Resources . 

2.5.5  Trade  Studies . 

2.6  T echnical  Management andControl . 

2.6.1  TechnicalBas  eline  Management  and  C  ontr  ol  ( Strategy  and  Approach) . 

2.6.2  TechnicalReviewPlan  (Strategy  and  Approach) . 

2 . 7  Integration  with  Other  Management  C  ontr  ol  Efforts . 

2.7.1  Ac  quisition  Strategy. . 

2.7.2  Risk  Management . 

2.7.3  Integrated  Master  Plan . 

2.7.4  Earned  Value  Management . 

2.7.5  Contract  Management . 
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N65236-593-PMP-0001-vl 


August  18,  2006 


m 


PROJECT  PLANNING 


CMMI®-  SE/S  W 

GoaL'Practice 

Number 


Compliance  matrix 
cross  references 
CMMI®  practices 
with  associated 
SSC-C  Process 
Manual  and  Project- 
specific  plan 

(No  matrix  for  SEP) 


1 


pp  l  .i 


PP  1.2 


PP  1.3 


PP  1.4 


PP  2 


CMMF-SE/SW  Level  2 
Process  Area 
Project  Planning  (PP) 


Est ab  li  s  h  E stim ate s .  Estim ate s  of  proj e  ct 
planning  parameters  are  established  and 
maintained. 


Estimate  the  Scope  of  the  Project.  Establish 
and  maintain  a  top-level  work  breakdown 
structure  (WBS)  to  estimate  the  scope  of  the 
project. 


Establish  Estimates  of  Project  Attributes. 
Establish  and  document  estimates  of  the 
attributes  of  the  work  products  and  tasks. 


Define  Project  Life  Cycle.  Define  the  project 
life  cycle  phases  upon  which  to  scope  the 
planning  effort. 


Determine  estimates  of  Effort  and  Cost. 
Estimate  the  project  effort  and  cost  for  the 
attributes  of  the  work  products  and  tasks 
based  on  estimation  rationale. 


Develop  a  Project  Plan.  A  project  plan  is 
established  and  maintained  as  the  basis  for 
managing  the  project. 


SSC-C 
PP  Process 
Manual 
Paragraph 


3.2 


3.2 


3.2 


3.2 


3.2 


3.3 


593 

PMP 

Paragraph 


1.2.1 


1.2.1 

3 

Appendix  A 


1.2.1 

1.3 


1 

1.2.1 


1.3 

1.2.1 

Appendix  A 


1 

1.2.1 
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•  Organizational  database  for  collecting  standard 
project  measures  and  providing  analysis 

*  Currently,  the  OMR  accepts  the  following  standard 
project  measures 


Category 

Core  Measure 

Schedule  Performance 

•  Estimated  vs.  Actual  Milestone  dates 

•  Estimated  vs.  Actual  Monthly  Task 
completions 

Cost  Performance 

•  Estimated  vs.  Actual  Milestone  costs 

•  Estimated  vs.  Actual  Monthly  costs 

Process  Performance 

•  Total  #  of  noncompliance  issues 
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OMR  Structure 
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(C __ 

z _ 

7 

OMR 

OMR  Client 

^  Datastore 

Application 

7 

"I 


Organizational  Performance 
&  Analysis 


Population  Size:  172 

Mean:  -21.2256 
Median:  -6.2256 
Mode:  0  .  0056 
Min:  -163.1256 
Max:  330.7756 
Variance:  71.8S56 
Standard  Deviation:  84.7656 
Probability  of  X  <  Min:  4.7SS6 
Probability  of  X  >  Max:  0  .  0056 
Probability  of  Min  <  X  <  Max:  9S.2S56 


(ALL)  Phases  of  (ALL)  Projects  Percentage  of  Schedule  Deviation 
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OMR  Application 


SP/MR 

7^ 


Repository)  Application  VI. 0.6 


Bl® 


Systems  Center 

Charleston 


*  Provides  interface  for 
input  and  query  functions 

*  Generates  quarterly 
organizational  report 

*  Projects  can  use  to 
manage  own  projects 

-  Capture  standardized  cost, 
schedule,  and  process 
performance 

*  OMR  implementation 
included  hands-on 
training 

*  Laying  the  groundwork  for 
higher  maturity 


File  Configuration  Manage  Help 


X 

Delete  Value 


Value:  Notes:  1 


Team  SPAWAR 
i  |C||j  ML3  Program 

cics 


15  Additional  CICS  Workstations  for  Distribution  Through  SSC-C 


Actual  Cost 
Estimated  Cost 
Actual  End  Date 
Actual  Start  Date 

-  Measured  Value:  8/6/2004 
1 1  j  iff  |  Measured  Date:  3/2/2006 
^  Measured  By:  CES 


J  Estimated  End  Date 
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,  2.1.2  Schedule  Variance 
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2.1.3  Process  Improvement] 

2.1. 3.1  All  Process  Areas 


Fupjl  it  1  tin  Sl2«:  5  - 

:  - .  - :  d 

wedlwv:  0.00* 
Hade:  .  ■ 

urn:  .6^* 

Bar :  ■ .  ■« if 

variable:  i 
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Probability  of  X  c  Hun:  <  •  * 

probability  of  X  >  v*. :  3S.M« 
Probability  of  ncn  ^  X  *  hm:  S-s  •  if. 
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SP/MR 


OMR  Reports 
Project-Level  Schedule  Deviation 
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Project  Phase  Schedule  Deviation 
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Phase.'Milestone 


Additional/Modified  Measures 
To  Be  Implemented  in  OMR 


SRWNl 

V 

Systems  Center 

Charleston 


Category 

Core  Measure 

Cost  Performance 
(More  granularity) 

•  Government  vs  Contractor  budget 

-  ODC 

-  T  ravel 

-  Training 

-  Materials 

Quality 

•  Peer  Reviews 

-  Effectiveness 

-  ROI  (hours  expended  vs  hours  saved) 

•  Pre-Deployment  Defect  Detection/Prevention 

-  Defect  decrease  for  successive  phases 

-  PITCO  vs  SOVT  defects 

•  Post-Deployment  Defects 

Need  improved  project  and  organizational  measures  to 
address  Maturity  Level  4/5  requirements 
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SP/MR 


Appraisal  Wizard  Tool 
Used  for  SCAMPI  Appraisals 


Systems  Center 

Charleston 


Designed  for 

CMMI 

appraisals 

Link  to  project 
documents 

Easy  to 
configure 

Captures  team 
comments 

Improves 
efficiency  of 
appraisal  team 


O,  Element  Review  (AM009)  Element:  PP  SP  2.1-i 

Options  Record  View  -  :  Document  Filtering 

t  Document  List  T  ^ 

_  Rating  Level  Appraisal 

t  pfc  Element  Colo 

■  Model  |CC11  *||  Element  Type  |Practice 

|  Record  Filter  |NONE 

View 

Appraisal 
>._/  Wizard 

v7 


Drag  a  column  header  here  to  group  by  that  column 


Drag  a  column  header  here  to  groc 


Element 

Pari 

PPSP  1.1-1 

PP 

PPSP  1.2-1 

PP 

PPSP  1.3-1 

PP 

PPSP  1.4-1 

PP 

PPSP  2.1-1 

PPSP  2.2-1 

F'F' 

PPSP  2.3-1 

PP 

PPSP  2.4-1 

PP 

iRecID  Record  Ty|  Status 

1 07  Compliant  Accepted 


Verifica  Record  Tent 


IDS  FOAMS  and  Budget  reports  are  direct  OE  ttW  uuuyi,^ 


□  CS  Schedule  FY2007  and  Budget  breakdown  for  OS  are  direct  OE.  Funding  summary  provides 


MS  Project  schedules  for  various  years  including  FY07  plan;  Requirements  Baselines  show  training 


Monthly  JTWS  Financial  Report  ■  March  2006  and  JTWS  Master  Schedule  are  direct  OE  that  budget 


mrl  C  i-ilnr-if-li  ilr-in  - 


Practice 
Characterizations 


icLQE.  Spend  plans  [05  and  06]  provides  budgets  [direct  OE). 


□ 


^ -  - ^ 

j  MM  <  >  wm  +  -  ^  S  x  X 

SP  2.1  -1  Establish  the  Budget  and 
Schedule 

Establish  and  maintain  the  project's 
budget  and  schedule. 

The  project's  budget  and  schedule  are 
based  on  the  developed  ^Nimates  and 
ensure  that  budget  allocati\ 
complexity,  and  task  depend 
appropriately  addressed. 


Record  Fields  Elements  /  Projects  /  T earn  Members  /  Data  Sources  |  Record  Documents  [ 
Create  New  Document 


Drag  a  column  header  here  to  group  by  that  column 


Individual  Project 
Records  per 
Practice 


Evidence  T  Doc  Type 


Comments  (Di 


Specific 

Practice 

Description 


Evidence  List  by 
Practice  &  Project 


Appraisal  Wizard  is  a  product  from  Integrated  Systems  Diagnostics,  Inc. 

http://www.isd-inc.com 


Te 


SEI  Partner 


CHSOFT 


Technical  SoftiArare  Services,  Inc. 


29 


Approved  for  public  release;  distribution  is  unlimited  (3  OCT  2007) 


SRWAR 

▼ 

Systems  Center 

Charleston  _ 

Training 


>  Training  Architecture 

>  Courses 


Technical  Software-  Services,  Inc. 


N65236-ENGOPS-BRIEF-0044-1 .3 


Approved  for  public  release;  distribution  is  unlimited  (3  OCT  2007) 


SE  &  PI  Training  Architecture 


SP/MR 


Systems  Center 
Charleston 


r 


Foundation  of 
PI  and  CMMI® 


■< 


PI  WBT 


SEI  Intro  to  CMMI® 

3-day 


r 


i 


Core  SSC-C  project 
and  engineering 
processes  a 
(Level  2  and  3) 


v 


Subject  Matter  Experts  - 
Use  commercially 
available  on-site  classes 


^  Engineering  Project  N 
&  Process  Mgmt 
v  Workshop  , 

(  ~\ 

SE 

Fundamentals 

V  J 

f  \ 

SE  for  Managers 

r  Intro  to  N 

Software 

v  Engineering  y 

SEMP  Workshop 

i 

SE  101  WBT 


Architecture  Dev.  WBT 


Risk  Management  WBT 


■< 


Quality  Engineering  Requirements  Analysis 


Risk  Management 


Prepare  Projects 
for  BSC  or  SCAMPI 


< 
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I  I  Existing  -  revise 

I  I  New  -  develop 

I  I  New  -  buy 
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Intro  to  Process  Improvement  WBT 


sFmm 
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SFWAR 

Introduction  to  SSC-C  Process  Improvement 


Refraining 

□  Courseware  Operations 

□  Course  Introduction 

□  Introduction  to  ..Process  Improve^m 

□  Terminology 


□  I.h.e  CMMI®  Model 

□  SS(>C  Irnplementatipn 

□  Organij^tional 


Originally  given  as  a  podium 
course,  converted  to  Web 
Based  Training  in  2004 


□  Process  Man 

□  CourseSummary 


Now  required  for  all 
employees 


SEIPar 
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SEI  Intro  to  CMMI®  for  SSC-C 


Systems  Center 
Charleston 


3-day  Introduction  to  CMMI®  course 
teaches  the  full  CMMI®  model 


aught  on's't®. 
since  Apr-  200 


-Students  learn  how  the  best  practices  build  and  relate 
across  process  areas 

-  Learn  the  terminology 


•SEI-Authorized  instructors  are  well-versed  in 
our  implementation  to  augment  material  with 
SSC-C  specific  content 

-  Highlight  SSC-C  tools  and  resources 

-Actively  involved  in  projects,  teams,  and  infrastructure 

•Over  350  employees  trained 

-Want  to  build  a  cultural  foundation  within  the 
engineering  departments 

SE  I  Part™ 
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Systems  Engineering  Training 


SP/MR 
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3-day  on-site,  classroom  course 

-  Based  on  SMU  SE  Masters  course 

-  Customized  to  incorporate  SSC-C  SE  process 

-  Over  340  SSC-C  engineers  trained 

1-day  SE  for  Managers  course  added 

-  Over  60  SSC-C  managers  trained 


“It  was  extremely  beneficial  to  have  a  professor  with  extensive 
knowledge  of  the  subject  matter  and  one  who  could  apply  it  to  the 
SPAWAR  methods.’’ 

“The  most  positive  aspects  I  took  from  the  class  was  the  visual 
correlation  with  what  was  asked  for  and  what  was  produced.  ” 

“I  would  recommend  it  to  all  the  program  leads/engineers.  ” 

Student  Feedback 
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New  On-Site  Courses 
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*  Risk  Management 

-  Piloted  in  September,  2007 

•  4-day  course 

-  Designed  for  Risk  Managers  or  Project  Managers 

*  Engineering  Project  &  Process  Mgmt  Workshop 
(aka  SE  Process  Improvement) 

-  Focus  on  how  to  use  the  SSC-C  processes  on  your  project 

•  Using  ePIan  Builder  to  develop  plans 

•  How  to  establish  your  CM  and  PPQA  procedures 

-  Round  2  of  curriculum  review  completed  in  September 

*  Quality  Assurance  (FY2008) 

-  Initial  discussions  held  with  ASQ  certified  instructor  to  tailor 
course  for  Quality  Managers  at  the  project  level 
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Web  Based  Training  (WBT)  Modules 
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Charleston 


*  Developed  to  directly  meet  SSC-C’s  needs 

-  Embedded  links  directly  to  SSC-C  documents  and  SOPs 

-  DAU  too  ACAT-level/large  program  oriented 

*  WBTs  feature  extensive  branching  and  rollovers 

-  Better  course  flow  and  maintains  interest 

-  Provides  more  detail  for  those  interested 

*  Audio  summary  on  many  pages 

*  Bookmark  progress  -  come  back  later 

*  Courses  developed  to  be  NMCI  and  508  compliant 

-  Utilize  HTML,  JavaScript,  and  ASP  pages  with  SQL  Server 
database 

-  Designed  for  Internet  Explorer  (5.5  +),  Flash  (5.0  +),  Windows 
Media  Player  (9.0  +) 

SElPsrtrw 
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SE  101  Web-Based  Training 
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Introduction  to  Systems  Engineering 

-  10-module  web-based  training  (-16  hours) 


Closely  aligned  to  SSC-C  SE  Process,  SE  Fundamentals  Course, 
ISO/IEC  15288  and  IEEE  standards 

Includes  hotlinks  to  referenced  documentation 

•  Process  manuals,  policies,  standards 

•  Great  for  Topic-specific  refresher  training 


ISO/IEC  152&8 
Technical 
Processes 


ClWIVf#®  se/sw 

Best  Practices 


Initial 

Capabilities 

Document 

(ICO) 

i 


/  /  it 

SYSTEMS  ENGINEERING  PROCESS 


i - 

Capability 

Development 

Document 

(ODD) 

— 

System 

Requirements 

Document 

(SRD) 

i 


| 

Ope  rati  on  a  [ 
Concept 
Description 
(OCD) 


5SC*C 

Standard  Processes 


Software 

Hardware 

Interface 

Requirements 

Requirements 

Requirements 

Specification 

Specification 

Specification 

(SRS) 

(HWRS) 

(IRS) 
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*  Topics 

-  Risk  identification 

-  Analysis  tools  and  techniques 

-  Mitigation  planning 

-  Risk  monitoring 

*  Section  Test  Questions 

*  Hot  Links  to  Examples 

-  SSC-C  Formats 

-  Project  Risk  Reports 

-  Tools 

-  DAU  /  External  resources 


Continuous  Risk  Management  Process 


Click  on  each  graphic  link  to  view  its 
description. 


More  relevant  and  understandable  for 
SSC-C  than  the  DAU  module 
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Architecture  Development  WBT 
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*  Introduction  to  Architecture  Development  and  DoDAF 

-  Designed  to  educate  and  promote  value  of  system  architecture  to 
non-architects  and  new  engineers 

-  Tests  for  understanding  after  each  section 
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What  We  Have  Accomplished 


SHMR 
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•  Process  Focus 

-  Defined  Policies  and  Processes 

-  Aligned  with  DoD  and  SPAWAR  guidance 

-  Aligned  with  industry  standards  and  CMMI®  model 

-  Built  organization  structured  around  processes  and  process  improvement 

•  Training  is  Critical 

-  Providing  Fundamentals  of  Engineering  for  new  and  old  professionals 

-  Developed  web-based  training  for  “self-paced”  and  refresher  training 

-  Defining  a  structured  technical  career  development  path  for  engineers 

•  Tools  for  the  Engineers 

-  Developed  ePIan  Builder  application  to  generate  planning  documents 

-  Developed  templates,  checklists,  and  web-based  document  repositories 
to  link  standards  and  DoD  guidance  to  day-to-day  tasks  and  processes 


Early  and  persistent  Systems  and  Software  Engineering 
applied  to  programs  and  projects 
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Lessons  Learned 
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•  Senior  Management  support  is  critical  to  success 

•  Training 

-  Everyone  needs  to  be  engaged  -  “train  the  masses” 

-  Specific  training  for  process  owners/subject  matter  experts 

•  Utilize  Teams  (IPTs)  as  champions  of  specific  processes 

-  Multi-department  representation 

-  Change  agent  mentality 

-  Process-focused  charters 

•  Resource  Properly 

-  Implement  with  projects  that  want  to  improve,  can  benefit  from  efforts, 
and  that  recognize  own  weaknesses 

-  EPO  staff  provided  skilled  coaching,  resources,  support,  and  tools 

-  Project  members  learned  by  doing  and  maintaining 

•  Goals  and  Publicity 

-  Keep  goals  to  sizable  bites  (projects) 

-  Publicize  successes;  Share  best  practices 

SElRsrtJ'rt 
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Is  the  SE  Revitalization  Working  ? 


shmr 

Systems  Center 

Charleston 

•  Recognition  of  SE  and  CMMI  effort 

-  1st  SPA  WAR  Systems  Center  to  achieve  Maturity  Level  2  (2005) 

-  1st  SPA  WAR  Systems  Center  to  achieve  Maturity  Level  3  (2007) 

-  Multiple  presenter  at  NDIA  SE  and  CMMI  conferences 

•  High  interest  in  Tools,  Training,  and  Implementation 


srmm 


World  Class 
Systems 
Engineering 

Capability 

Maturity 

Model 

Integration 

(CMMI®) 


Systems 

Center 

Charleston 


MATURITY  LEVEL 

\  April  28,  2005 


SYSTEMS 


srmm 


Capability 

Maturity 

Model 

Integration 

(CMMI®) 


MATURITY 

LEVEL 

l  April  27, 2007 
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Is  the  SE  Revitalization  Working  ? 
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•  Business  Results 

-  SCN:  “They  see  us  as  a  model  and  want  to  increase  our  efforts.” 

-  Automation  Program:  “We  had  hundreds  of  sites  and  there  was  a 
need  for  a  structured  organization  to  put  a  ‘wrapper’  around  that 
and  control  it.  CMMI  became  the  wrapper.” 

-  CICS:  “CMMI  was  key  to  achieving  the  project  goal.” 

-  VIDS:  “The  VIDS  failure  (2000)  motivated  implementing  CMMI 
because  the  team  needed  to  change  course  or  the  customer 
would  have  no  confidence  in  system  development.  It  was  a 
tremendous  success...” 

*  Others  Asking  for  Help 

-  PMS  408  -  CREW  program 

-  SESG  /  NAVAIR  /  NAVSEA 

-  Marine  Corp  -  Quantico 

-  Air  Armament  Center,  Eglin  AFB 
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Going  Forward 
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Systems  Center 

Charleston 

*  Increase  usage  of  tools  across  departments/projects 

*  Add  additional  plans  to  ePIan  Builder  as  needed 

*  Continue  internal  CMMI  Level  3  mini  assessments 

*  Enhance/Expand  OMR 

*  Command  and  Department  Project  Reviews  process 

-  Look  at  quality  of  plans  and  implementation  of  best  practices 

-  Reviews  of  project  status  by  management  driven  by  project 
metrics 

-  More  Peer  Reviews  to  measure  “saves” 

*  Better  tailoring  guidance  for  smaller  projects 


Begin  Maturity  Level  4/5  implementation 
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Systems  Center 
Charleston 


Any  Questions? 


Contact  Information: 

Michael  T.  Kutch,  Jr. 

SPAWAR  Systems  Center  Charleston 
Email:  michael.kutch@navv.mil 
Phone:  843-218-5706 


Mike  Knox 
TECHSOFT,  Inc. 

Email:  miknox@techsoft.com 
Phone:  850-469-0086 


Iechsoft 


Techni  cal  Software-  Services,  Inc. 


N65236-ENGOPS-BRIEF-0044-1 .3 


Approved  for  public  release;  distribution  is  unlimited  (3  OCT  2007) 
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Applying  Design  of  Experiments 
(DOE)  methodology  to  Sortie 
Generation  Rate  (SGR)  Evaluation 


Design  of  Experiments 
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Josh  Tribble 

MILITARY  ANALYST 
AVW  TECHNOLOGIES 


Phone:  757-361-9587 
E-mail:  tribble@avwtech.com 
860  Greenbrier  Circle,  Suite  305 
Chesapeake,  VA  23320 
http://www.avwtech.com 
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Agenda 


Introduction 

-  Acquisition  humor 

-  The  Integrated  T&E  Challenge 

Intro  to  Design  of  Experiments 

SGR  Assessment  Methodology 

-  Overview  of  SGR  Assessment  to  date 

-  SGR  Assessment  objectives,  MOEs,  factors 

-  SGR  Testbed  Assessment  Design  Factors  /  Run  Matrix 

-  SGR  Live  Testing  Validation 

Benefits  of  DOE  over  single  scenario  based  analysis 
Conclusion  /  Q&A 


NOTE:  My  remarks  are  intended  to  spur  thought  on  improving  how  we  as  testers  can  do 
business  better  to  support  the  warfighter.  While  I  hope  this  aligns  well  with  DoD  and 
Services  T&E  initiatives,  I  am  not  representing  any  government  agencies’  official  position. 
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Acquisition  101? 


How  the 
user  described  it 


How 

the  requirement 
was  understood 


How  the 
contractor 
designed  it 


How  the 
programmer 
wrote  it 


How  the 
PM/sponsor 
described  it 


How  the  project 
was  documented 


What  was 
actually  installed 


How  the 

Government  was 
billed 


■ 

How  the  helpdesk 
supported  it 


What  the  user 
really  needed 


How  do  we  avoid  this? 


Integrated  T&E  Challenge 

•  Coordinated  planning  and  development  of  individual  test 
objectives 

DT  /  CT  /  OT  /  LFT&E  remain  separate  but  leverage  data 
and  resources  whenever  possible 

Potential  for  significant  cost  savings  and  earlier  risk 
reduction 

Requires  buy-in  from  all  orgs  +  strong  T&E  Working  IPT 

Requires  strong,  up-front,  test  planning  and  data  analysis 
methodology  -  Design  of  Experiments  (DOE!} 


Wf  OT&E 

kl|  DT&E 

CT 

LFT&E 


4 


Integrated  T&E 


r 

integrated  /  ~ 

^  P 


System  Disposal 

f  (CT,  DT,  OT,  LFT&E,  Joint  Exp,  M&S,  Analysis,  etc.)  dt 

Program  Conception 
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Background  of  DOE 


DOE  originated  in  the  field  of  agricultural  studies  in  the  1930s  by  R. 


Fisher,  building  on  W.T.  Gossett’s  work  at  Guinness  Brewery — Brilliant! 
Used  throughout  industry  in  industrial  experiments,  process  improvement, 
statistical  process  control 

USAF  has  significant  experience  in  use  of  DOE  across  numerous 
programs;  Navy  is  beginning  to  implement 

DOE  methodology  is  used  to  interrogate  a  process,  improve  knowledge  of 
how  the  process  works,  and  identify  factors  and  interactions  affecting 


variability  of  performance  outcomes. 


AVW 


DOE  Process  Goal  /  Benefits 


Compared  to  other  systematic  methods  DOE  designs: 


Design  of  Experiments 


u 


Yield  better  process  understanding 
Can  be  planned  and  analyzed  faster 
Cheaper  -  using  between  20-80%  of  usual  runs/tests/resources 
Better  exploration  across  range  of  performance — depth  and  breadth 
of  testing 

Challenge  assumptions  and  demonstrate  real  performance 
Better  way  to  design  and  test  complex  systems 


AVW 
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DOE  Process  Outlin 

4  Basic  Steps 


Project  description  and  decomposition 

•  Problem  statement  and  objective  of  experiment  (test) 

•  Response  variables,  and  potential  causal  variables  -  Ishikawa  fish  bone. 

Plan  test  matrix 

•  Determine  constraints,  prioritize  factors,  and  select  statistical  design  (2Kvs.  3Kvs.  mixed, 
Taguchi  vs.  classical  arrays,  full  vs.  fractional,  non-linear  effects?,  replications?,  blocking?) 

•Write  the  test  plan  with  sample  matrices,  profiles,  and  sample  output;  run  sample  analysis. 
Produce  observations  -random  run  order  &  blocked  against  unknown  effects 

•  Block  runs  to  guard  against  uncontrollable  unknown  effects  as  needed. 

Ponder  the  results 


Analyze  and  project  data;  draw  conclusions,  redesign  test  as  necessary  and  assess  results. 
Perform  “salvo  testing”  (test-analyze-test);  screen  large  #  of  factors  then  model 
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SGR  Assessment 
Methodology 
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SGR  Assessment 

Reguirements 


•  SGR  Key  Performance  Parameter 


THRESHOLD 

OBJECTIVE 

Sustained 

SGR 

Average  of  160  operational  combat 
equivalent  aircraft  sorties  in  12  hours 
of  launching  per  day  over  30  days  (26 
Flying  and  4  Non-Flying  Days  as 
specified  in  the  Design  Reference 
Mission  (DRM)  -  total  cycle  of  4160. 

Average  of  220  operational  combat 
equivalent  aircraft  sorties  with  12  hours 
of  launching  per  day  sustained  over  30 
days  (26  Flying  and  4  Non-Flying  Days 
as  specified  in  the  DRM)  -  total  cycle  of 
5720. 

Surge  SGR 

(requires  crew 
augment) 

Average  of  270  operational  combat 
equivalent  aircraft  sorties  generated 
during  each  successive  24-hour 
period  over  4  continuous  days. 

Surge:  average  of  310  operational 
combat  equivalent  aircraft  sorties 
generated  during  each  successive  24- 
hour  period  over  4  continuous  days. 

Other  Measures  of  Performance:  cycle  times,  task  timing, 
launch  and  recovery  cycles,  resource  usage,  crew  fatigue  levels,  fuel 

states/rates,  etc. 
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SGR  Assessment  Testbed 


M&S  testbed  captures  times  and  actions  associated  with  preparing, 
launching,  and  recovering  sorties  per  the  DRM 


X 
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Revdlf  PtadwclctD  Ship 
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froces^rndmE 


M&S  matured  and  validated  over  time  prior  to  runs  for  score 

Live  test  used  for  validation  once  ship  is  delivered  and  aviation  certified 


SGR  Model 
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-  Mission  Planning  Timing 

-  Aircraft  Recovery  Time  Which  Encompasses: 

•  Fueling  Time 

•  Ordnance  Handling  Times 

•  Aircraft  Movement/Spotting  Times  On  The  Flight  Deck 

•  Aircraft  Movement/Spotting  Times  In  The  Hangar  Bay 

•  Aircraft  Availability 


SGR  Assessment  Analysis 

Objectives 

Determine  average  SGR  over  DRM  to  meet  KPP  requirement 
Determine  active  factors  influencing  the  variability  &  overall  outcome 

-  Measure  %  sorties  completion  rather  than  binomial  pass/fail 

-  Each  day  in  the  DRM  treated  as  a  single  design  point  due  to 
interdependencies  of  events  within  that  day 

Provide  the  fleet  with  an  analytical  model  showing  probability  of  meeting  a 
given  airplan  based  on  its  size,  mission  composition,  environment,  and 
any  other  active  factors 


%  Airplan  _  Sorties  _  Completed 


Daily  _  sorties  _  completed  _  succesfull  y 
Airplan  _  sorties 


xl00% 


Allows  equal  comparison  of  the  4  T/O  surge/sustained  requirements  across  all 
factors 

Continuous  dependent  variable  provides  more  statistical  power  than  pass/fail 
Supports  more  robust  assessment  of  capes  and  lims 


SGR  Factor  Selection 


TECHNOLOGIES 


Experimental  control  factors : 

•  Environmental 


-  Visibility/Sky  Cover:  Clear  Skies  (Case  I)  or  Clouay/iNigni  ^uase  imj 

-  Time  of  day:  midday  or  midnight  (for  12  hour  ops,  N/A  for  24  hour  ops) 


-  Sea/Winds:  state  1  vs.  3 


•  Systems: 

-  Availability:  100%  &  actual  (for  CVN-21  systems  and  aircraft) — allows  for  analysis  of 
impact  of  equipment  failures 

•  Mission 

-  Sortie  Size:  Threshold  &  Objective  levels  from  the  DRM 

-  Sustained  and  Surge  Mission  (12  vs.  24  hr  ops  (with  augmented  crew)) 

-  Operation  day:  early  and  late  in  ship  on-station  operational  period;  expect  to  interact 
with  availability  for  system  failures  and  also  translates  to  possible  crew  fatigue 

-  Airplan  mission  mix:  early/late  DRM  days  representing  different  ordnance  mix; 

-  Mission  mix  and  operation  day 


SGR  Factor  Selection  (cont’ ) 

ContmlJable  Factors  field,  constant 

•  Underway  Replenishment 

-  Not  a  factor  of  SGR  but  presumed  to  occur  on  assigned  days  or  fuel  and  ordnance  will 
not  be  available  for  the  planned  flight  days) 

•  Aircrew  augmentation 

-  Confounded  with  mission  type  -  assumed  normal  crew  for  sustained  operations  and 
augmented  crew  for  surge  missions 

Measurable  Noise  Factors 

•  Other  environmental  factors  not  controlled  (if  in  test  /  model) 

-  Temperature  extremes 

•  Specific  metrics  in  the  subordinate  models  driven  by  the  main  inputs,  such  as: 

-  Crew  fatigue  (driven  by  the  mission  day) 

-  Resource  availability 

-  Number  of  aircraft  available 

-  Weapon  skids  available 

-  Timing  for  critical  tasks,  etc. 
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SGR  Factor  Selection  (contl 
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Design  factors: 

-  Factors  with  highest  expected 
influence  listed  first 

•  Important  when  setting  up  fractional 
factorial  matrices — usually  easier  to 
resolve  factors  and  interactions 

-  Setup  for  M&S  only;  cannot  test  all 
of  these  in  live  testing 

-  Requires  M&S  improvements 

-  Need  buy-in  for  “excursions”  above 
threshold 

•  High  levels  force  the  “system” 
towards  a  higher  failure  rate  to  see 
more  variation  in  response 


Setting 

Factor 

(Low) 

-1 

(Center 

Point) 

0 

(High) 

+1 

A 

Surge/ 

Sustained 

Operations 

Sustaine 
d  (12  Hr 
ops) 

N/A 

Surge  (24 

Hr  ops 
w/augment) 

B 

Sortie  Size 
(T/0) 

Thres¬ 

hold 

Halfway 

btwn 

Objective 

C 

operational 

day 

Early  (1/4 
or  5/30) 

Mid  (2/4 
or  15/30) 

Late  (4/4  or 
26/30) 

D 

Availability 

100% 

Halfway 

btwn 

actual  /  spec 

E 

Visibility/ 

Cloud 

Cover: 

Clear/ 
Case  1 

Partly 
Cloudy/ 
Case  II? 

Cloudy/ 

Case  III 

F 

Seakeeping 

motion 

effects 

5  kts/SSI 

12 

kts/SS2 

20  kts/SS  3 

G 

Time  of  day 

Day 

Dusk? 

Night 

H 

Mission 

Day 

Early 

Mid 

late 
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SGR  Testbed  Run 
Assessment  Design 


Full  factorial  requires  28  or  256  runs 

-  Unnecessary  since  many  effects  are 
inactive 

Resulting  test  matrix  is  a  resolution  IV 
28-4  fractional  factorial  of  16  runs  +  8 
additional  runs  for  central  composite 
design 

-  Some  interactions  are  confounded  but 
can  be  resolved 

Model  DRM  days  per  the  assigned 
settings  and  evaluate  SGR  Compl  % 

“salvo  test”: 

-Runs  1-8,  then  analyze  for  effects 

-Runs  9-16,  then  reanalyze  for  effects 

-Perform  center  points  to  check  for 
linearity 

-If  necessary,  run  CCD  (face  points)  for 
non-linear  effects 


Run 

Blk 

A 

B 

c 

D 

E  = 

ABD 

F= 

ACD 

G= 

BCD 

H= 

ABC 

1 

Factorial 

1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

2 

Factorial 

1 

-1 

-1 

-1 

+1 

+  1 

+  1 

+  1 

-1 

3 

Factorial 

1 

-1 

-1 

+1 

-1 

-1 

+  1 

+  1 

+  1 

4 

Factorial 

1 

-1 

-1 

+1 

+1 

+  1 

-1 

-1 

+  1 

5 

Factorial 

1 

-1 

+  1 

-1 

-1 

+  1 

-1 

+  1 

+  1 

6 

Factorial 

1 

-1 

+  1 

-1 

+1 

-1 

+  1 

-1 

+  1 

7 

Factorial 

1 

-1 

+  1 

+1 

-1 

+  1 

+  1 

-1 

-1 

8 

Factorial 

1 

-1 

+  1 

+1 

+1 

-1 

-1 

+  1 

-1 

9 

Factorial 

2 

+  1 

-1 

-1 

-1 

+  1 

+  1 

-1 

+  1 

10 

Factorial 

2 

+  1 

-1 

-1 

+1 

-1 

-1 

+  1 

+  1 

11 

Factorial 

2 

+  1 

-1 

+1 

-1 

+  1 

-1 

+  1 

-1 

12 

Factorial 

2 

+  1 

-1 

+1 

+1 

-1 

+  1 

-1 

-1 

13 

Factorial 

2 

+  1 

+  1 

-1 

-1 

-1 

+  1 

+  1 

-1 

14 

Factorial 

2 

+  1 

+  1 

-1 

+1 

+  1 

-1 

-1 

-1 

15 

Factorial 

2 

+  1 

+  1 

+1 

-1 

-1 

-1 

-1 

+  1 

16 

Factorial 

2 

+  1 

+  1 

+1 

+1 

+  1 

+  1 

+  1 

+  1 

17 

Center  rep  1 

3 

-1 

0 

0 

0 

0 

0 

0 

0 

18 

Center  rep  2 

3 

-1 

0 

0 

0 

0 

0 

0 

0 

19 

cd  face  point  -b 

4 

-1 

-1 

0 

0 

0 

0 

0 

0 

20 

cd  face  point  +b 

4 

-1 

+  1 

0 

0 

0 

0 

0 

0 

21 

bd  face  point  -c 

4 

-1 

0 

-1 

0 

0 

0 

0 

0 

22 

bd  face  point  +c 

4 

-1 

0 

+1 

0 

0 

0 

0 

0 

23 

be  face  point  -d 

4 

-1 

0 

0 

-1 

0 

0 

0 

0 

24 

be  face  point  +d 

4 

-1 

0 

0 

+1 

0 

0 

0 

0 
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SGR  Ljve  Testing,  Validation 

Test  Design 


Live  test  conditions  and  cost  (potentially  $100M?)  limit  amount  of  live  test 
and  the  conditions 

Focus  on  validating  specific  test  points  of  interest  and  confirm  within  the 
M&S  runs  for  score 


Factor 

-1 

0 

+1 

Rationale 

A 

Surge/  Sust.  Ops 

Sustained 

N/A 

Surge 

Both  operations  can  be  run 

B 

Sortie  Size  (T/0) 

Threshold 

(T+  0)1  2 

Objective 

A  mix  of  sortie  sizes  can  be  run 

c 

Operational  day 

Early 

Mid 

Late 

No  means  of  imposing  a  late  day  due  to  cost 

D 

CVN-21/A/C  Ao 

100% 

Halfway 

Actual 

Actual  equipment  Ao 

E 

Cloud  Cover 

Actual  conditions? 

F 

Sea-State 

Actual  conditions? 

G 

Time  of  day 

Actual  condil 

ions? 

H 

DRM  Mission  mix 

Early 

j  Mid 

Late 

Factor  is  probably  inactive  so  randomly  assign 

SGR  Live  Testing  Validation 

Test  Design  (coni’) 


•  Final  Test  Matrix  with  settings: 


Test 

Case 

A:  Ops 
Type 

B:  Sortie  Level 

Actual  (# 
Sorties) 

H:  DRM 
Mission  Day 

Notes 

1 

Sustained 

Threshold 

160 

5 

Priority 

2 

Sustained 

Objective 

220 

26 

Priority 

3 

Surge 

Threshold 

270 

26 

Priority 

4 

Surge 

Objective 

310 

5 

Priority 

5 

Sustained 

Halfway  btwn 

190 

15 

Additional  run  for  midpoint 

6 

Surge 

Halfway  btwn 

290 

15 

Additional  run  for  midpoint 

7 

Sustained 

Threshold 

160 

26 

Additional  run  for  alternate  mission  mix 

8 

Sustained 

Objective 

220 

5 

Additional  run  for  alternate  mission  mix 

•  Recommend  run  during  Joint  Task  Force  Exercise  to  ensure  combat 
ready  crew  &  systems 

•  Some  analysis  of  variance  can  be  run  directly  but  main  objective  is  to 
compare  day  for  day  with  M&S  results  (including  V&V  of  lower  level 
measures  within  the  specific  process  models) 

•  Runs  1-4  are  priority;  select  additional  runs  based  on  M&S  results 
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SGR  Testbed  Assessment 

Sample  Data  Analysis 


Response  surface  plot  across  factors 
of  interest  showing  response  & 
interactions 

Table  of  plan  vs.  predicted  actual  SGR 
Completion  Rate  for  factor  settings  of 
interest  --  shows  SGR  completion  % 
falling  off  as  too  many  are  sequenced 

demonstrates  how  analysis  can 
describe  ship  caps  &  lims,  not  just  a 
pass/fail  grade  for  a  KPP  tested  only 
to  threshold 


0.0 

a 
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Planned  Sorties 
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CONCLUSION 


DOE  methodology: 

-may  significantly  reduce  the  required  runs  for  Testbed  Assessment  and  live  test 
validation  while... 

-providing  a  more  robust  process  for  statistical  analysis  of  variance  to  determine  where 
the  ship  design  can  and  cannot  support  a  given  air-plan  under  the  other  conditions 

-supports  robust  &  efficient  integration  ofJ/l&S  development  testing^  W&A,  & 
evaluation 


Design  of  Experiments 

m  t: — #  — »■, 
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•  DOE  is: 

-a  smarter  way  of  doing  testing 

-can  provides  superior  knowledge  to  the  systems  engineers 
-something  ail  testers  &  systems  engineers  should  become  familiar  with i 

•  QUESTIONS? 
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NDIA  Conference  23  Oct  2007 


*  The  Players-Users,  Acquisition  Team,  &  Industry 

*  The  Environment-CDD  &  Contract 

*  The  Grade-Financial  &  Performance 

*  Why  We  Are  Here 

*  Current  Challenges  To  Success 

*  The  Death  of  Acquisition  Reform 

*  Three  Phase  Procurement  Initiative 


SLIDE  2 


NDIA  Conference  23  Oct  2007 


The  Players 


SLIDE  3 


NDIA  Conference  23  Oct  2007 


The  Environment 


SLIDE  4 


NDIA  Conference  23  Oct  2007 
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The  Grade 


SLIDE  5 


High  Visibility  Programs 

NDIA  Conference  23  Oct  2007 


•  Army  And  Navy  ACS  Program 

•  Comanche 

•  Littoral  Combat  Ship  (LCS) 

•  Presidential  Helicopter  (VH-71) 

•  Coast  Guard  Deep  Water  Program 

•  Army  Future  Combat  System 

•  Launch  &  Recovery  (EMALS  &  AAG) 

•  Seal  Delivery  Vehicle 

•  CVN-78  USS  Gerald  Ford 


SLIDE  6 


NAVAIR  Burning  Platforms 


NAV^AI  R 
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Stressors  To  Current  SE  Process 


I  R 


NDIA  Conference  23  Oct  2007 


Inadequate  “Pre  Systems  Acquisition”  Phase 

-  CONOPs,  OPSITs,  TACSITs,  Modeling  &  Simulation 

-  Industry  Involvement  Required  To  Bound  CDD 

Specifications  Lack  Clarity 

-  Performance  Based  Specs 

-  Design  &  Certification  Standards-Non-Tailorable  Requirements 

-  COTS/ND  I 

-  Strategic  View  of  Life  Cycle  Cost 

Programmatic  Stressors 

-  Cost  And  Schedule  Set  At  Contract  Award 

-  Award  Fee  Schedule  Promoting  Bad  Behavior 

-  Lack  Of  Early  Sub-Contractor  Involvement 

-  Government  System  Development  Oversight 
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Acquisition  Reform  Retreat 


I  R 
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Acquisition  Reform  Tenets  Proven  WRONG 

-  Broader  Performance  Based  Specifications 

•  Required  Design  Standards  Missing 

•  Certification  Standards  Missing 

-Less  Government  Involvement 
-Contractor  And  Government  Goals  Aligned 
-90  Percent  Solution  Post  CDD  Is  Acceptable 
-CDRLs  And  Documentation  Not  Required 
-Risk  Identification  Is  A  Management  Tool 
-Award  And  Incentive  Fees  Motivate  Contractors 
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New  Approach 

NDIA  Conference  23  Oct  2007 


*  Clear  Requirements 

-  Industry  Involvement  In  Requirements  Development 

-  Design,  Build,  And  Certification  Standards 

-  Derived  And  Correlated  Requirements 

-  Use  Case  Analysis 

*  Contract  Governs  Communication 

-  Objective,  Deliverable  Evidence  To  Support  Oversight 

-  SEP  Issued  With  RFP 

-  Enforce  “Event  Driven”  Design  Maturity  To  A  Schedule 

-  IMS  &  EVM  Is  Essential  Forum  For  SE  Management 

*  Government  Acts  As  Prime  Integrator  Role 

*  Reality  Based  Budget  And  Schedule 


R 
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Design  Review 

Baseline 

SRR-System  Requirements 

Performance 

SFR-System  Functional 

Functional 

SSR-Software  Specification 

Software  Requirements 

PDR-Preliminary  Design 

Physical 

CDR-Critical  Design 

Build 

IRR-Integration  Readiness 

Integration 

TRR-Test  Readiness 

Test 

FRR-Flight  Readiness 

Airworthiness 
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A 


PHASE  I 


PHASE 


A 


PHASE 


MS-A 


MS-B 


RFP 


RFP 


SRR 


SFR 


RFP  PDR 


CDR 


Objectives/Process 


Objectives/Process 


Objectives/Process 


-Multiple  Contractors  Engaged 
-Develop  CDD  Through  JROC 

-Understand  Non-Tailorable 
Specifications 

-Assess  Technology  TRL  6 

-Develop  CONOPS,  OPSITs,  & 
TACSITS 

Output 

-Reasonably  Bounded  CDD 
-Formal  CONOPS 
-USE  CASE  Analysis 
-Conceptual  Solutions 
-Cost/Schedule  Cut 


-Down  Select  To  Few  Primes 

-Derive  System  Development 
Specification 

-Finalize  Tailorable 
Specifications 

-Apply  USE  CASES  To  Derive 
Funtionality 

-Develop  Interface  Specifications 
-Engage  Major  Sub-Contractors 

Output 

-Fully  Derived  System 
Development  Specification 

-Sub-System  Specifications 
-Models  &  Prototypes 

-IBR  Quality  Cost  &  Schedule 
Estimates 

-IMS  And  Initial  EVM  System 


-Down  Select  To  Single 
Contractor 

-Traditional  SDD  Process 
-Reduced  Risk  Environment 
-Derived  Requirement  Define 

-Realistic  Cost  And  Schedule 
Constraints 

Output 

-Proceed  To  Normal  Milestone  C 
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Summary 

NDIA  Conference  23  Oct  2007 


*  Revitalization  Goal 

-  Better  Requirements  Understanding  And  Stability 
-Better  Budget  And  Schedule  Discipline 

*  Additional  Phase  With  Multiple  Contractors 

-Supports  Bounding  Of  CDDs 

*  Control  Program  Inertia  Into  Milestone  B 

*  Implementing  Contracting  Strategies  Still  In 
Review 
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Software  Engineering  Institute 


Complex  Systems  of  Systems: 
The  Double  Challenge 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

Philip  Boxer,  Lisa  Brownsword, 
Ed  Morris 
23  October  2007 


Carnegie  Mellon 


©  2007Carnegie  Mellon  University 


Briefing  Objectives  and  Agenda 


Instigate  an  alternative  way  of  viewing  systems  of  systems 

•  Begin  equipping  participants  to  ask  different  questions  about  the 
challenges  and  the  opportunities 

Agenda 

•  Describe  a  project  approach 

•  Explore  implications  of  a  changing  world 

•  Describe  an  alternative  reasoning  framework 


-  Software  Engineering  Institute 


Carnegie  Mellon 
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Many  Organizations  Have  These  Problems 


The  DoD 

Other  federal  agencies 

Large  and  small  industrial  organizations  across  the  globe 

Recent  studies  by  the  SEI  and  international  consortia  show  that  large, 

systems  of  systems  (SoS)  are  endemic 

•  SoS  challenge  the  capabilities  of  high-performing,  high-capability 
organizations  accustomed  to  large  systems. 

•  These  challenges  surface  throughout  development,  acquisition, 
deployment,  and  evolution. 

•  These  challenges  derive  from  working  across  multiple  enterprises  in 
response  to  rapidly  changing  and  unanticipated  forms  of  operational 
demand. 


-  Software  Engineering  Institute 


Carnegie  Mellon 


The  Double  Challenge 

©  2007  Carnegie  Mellon  University 


3 


Creating,  Using,  and  Evolving  Composites  of 
Systems 


-  Software  Engineering  Institute 


Carnegie  Mellon 


The  Double  Challenge 

©  2007  Carnegie  Mellon  University 


4 


Why  isn’t  This  Straightforward? 


A  typical  approach 

•  Look  at  the  software  aspects  of  individual  systems 

•  Determine  which  ones  are  “good”  for  the  composite  system  of  systems 

•  Determine  how  to  put  the  good  ones  together — quickly 


-  Software  Engineering  Institute 


Carnegie  Mellon 


The  Double  Challenge 
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What  is  Needed:  A  Concept  of  “Operational” 
that  Takes  a  Broader  View 


Looking  at  the  Situation  from  a 
System-of-Sy stems  Perspective 


User  and  Other  Stakeholder 
Communities 


What  are  the  gaps 
between  what 
capabilities  are 
provided  (supplier 
push)  that 
responds  to  user 
and  stakeholder 
needs  ( operational 
pull) 


‘the  hole-in-the-middle” 


-  Software  Engineering  Institute 


Carnegie  Mellon 
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Key  Challenge:  How  Entities  Work  Together  and 
Resolve  Conflicts 


Number,  type,  and  roles  of  participants  are  increasingly  diverse,  reflecting 
differing  vested  interests 

Scarce  resources  and  the  need  for  concurrent  uses  make  a  single  decision 
authority  increasingly  unlikely 


Single  Task 

Svstem 


A  single  program  directs 
composition;  little  potential 
for  conflict 


Single  Enterprise 
System 


A  real  or  virtual  entity 
directs  how  multiple  entities 
collaborate  to  compose 
multiple  programs;  and 
resolves  potential  conflicts 
by  imposing  constraints 


Multi-Enterprise 


Multiple  real  or  virtual 
directing  entities  making 
competing  demands  on 
SoS;  and  conflict  resolution 
requires  negotiating  mutual 
constraints 


Category  names  from  “Architecting  Principles  for  Systems  of  Systems by  Mark  W.  Maier.  http://www.infoed.com/open/papers/systems.htm 


-  Software  Engineering  Institute 


Carnegie  Mellon 
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Key  Challenge:  Increasingly  Turbulent 
Operational  Contexts 


•  Customers  and  users  want  specialized  solutions  in  ever  shorter  time  frames 
continuously  adapted  to  their  changing  and  evolving  situations 


Suppliers  and  systems  have  to  become  more  agile  to  respond 


Solution-based 


comer 
Experience-based 


Users  want  products  or 
services  that  can  be 
provided  in  a  way  that  is 
unaffected  by  how  they  are 
used 


Users  want  integrated 
solutions  that  are 
customized  to  their  context, 
but  in  a  way  that  can  be 
specified  beforehand 


Users  want  integrated 
solutions  that  are 
customized  in  ways  that 
change  and  evolve 
throughout  the  life  of  the 
mission  that  they  support 


‘Turbulence’  as  per  “The  Causal  Texture  of  Organizational  Environments”,  Emery  F  E  and  TristE,  Human  Relations  1965,  18,  pp  21-32. 
Categories  adapted  from  “The  New  Frontier  of  Experience  Innovation”,  Prahalad  and  Ramaswamy,  MIT  Summer  2003 


-  Software  Engineering  Institute 
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A  Double  Challenge:  Diversity  of  Participants 
with  Turbulent  Usage  Contexts  and  Needs 


Source:  The  Double  Challenge, 

Philip  Boxer,  2006; 

http://asymetricdesign.  com/archives/1 6 


1  -  Collaborating 
Effectively  Across 
Boundaries 


Disruption  due 

to  addressing 
the  multi-enterprise 

governance  context 


Comfort  sons 
for  traditional 
engineering 


Disruption 
due  to 
diverse 
demands 
arising  from 
dynamic 
contexts 
use  co 


2-  Developing 
flexible 
responses  to 
changing 
situations 


-  Software  Engineering  Institute 
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The  Need:  Leveraging  the  Double  Challenges 


Multiple  forms  of  (potentially  non  pre¬ 
determined)  operational  effects 


integration 


Tntrffm 


the  multi-enterprise 
governance  context 


Collaborating 
across  boundaries 
to  provide  flexible 
responses  to 
changing 
situations 


Comfort  sone 
for  traditional 
engineering 


Disrupl 
due  to 
diverse 
deman 
arising 
dynam 
contexi 
use  coi 


-  Software  Engineering  Institute 
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An  Example 


Comfort  zone 
for  traditional 
engineering 


Disruption 
due  to 
diverse 
demands 
arising  from 
dynamic 
contexts  of 
use  context 


Disruption  due 
to  addressing 
the  multi-enterprise 
governance  context 


Where  were  they? 

Where  did  they  need  to  be? 
What  were  the  gaps? 


-  Software  Engineering  Institute 


Carnegie  Mellon 
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The  Situation 


Multi-national  stakeholders  in  an  acquisition  program  updating  a 
system  of  systems  within  an  operational  capability 

•  Operational  capability  itself  occupies  a  key  role  interoperating  with 
other  capabilities  within  a  single  unified  command  undertaking  joint 
missions. 

•  Issue  was  the  sustainment  of  the  operational  capability  through  its  life 
given  anticipated  changes  in  its  role  and  the  complex  nature  of  its 
systems. 

This  involved  three  challenges: 

1 .  managing  the  process  of  upgrading  within  the  context  of  sustaining  the 
operational  capability 

2.  improving  the  way  these  processes  are  managed  through  the  life  of 
the  capability,  given  their  systems-of-systems  nature 

3.  improving  the  role  of  acquisition  in  support  of  this  kind  of  sustainment 


-  Software  Engineering  Institute 


Carnegie  Mellon 
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Modeling  the  Whole  Space 


Objective:  understand  and  analyze 
technical,  cognitive,  process,  and 
organizational  elements  and  their 
inter-relationships — within  their 
context-of-use 


Disruption 
due  to 
diverse 
demands 
arising  from 
d^Qfirole^ 
confsxta-ei. 
use  context 


Structure-function 
and  trace  layers 


Disruption  due 
to  addressing 
the  multi-enterprise 
governance  context 


Comfort  zone 


Synchronisation 

layer 


Hierarchy  layer 


Demand  layer 


D 


-  Software  Engineering  Institute 
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5  Layers  of  Analysis 


Structure/Function:  The  physical 
structure  and  functioning  of  resources 
and  capabilities 

Trace:  The  digital  processes  and 
systems  that  interact  with  the  physical 
processes 

Hierarchy:  The  formal  hierarchies 
under  which  the  uses  made  of  both  the 
physical  and  the  digital  are  held 
accountable 

Synchronization:  The  lateral  relations 
of  synchronization  and  orchestration 
within  and  between  the  organizations 
providing  services  “on  the  ground” 

Demand:  The  nature  of  the  contexts-of- 
use  giving  rise  to  demands  on  the  way 
the  operations  are  organized  to  deliver 
effective  and  timely  services 


-  Software  Engineering  Institute 


Carnegie  Mellon 


The  Double  Challenge 

©  2007  Carnegie  Mellon  University 


15 


The  Outputs 


Stratification  analyses  different  levels  of  interoperability*  from  the  point  of 
view  of  the  demands  placed  on  the  system  of  systems  by  the  environment 


•  Synchronization  (Can  the  configurations 
needed  interoperate  in  practice?) 

•  Orchestration  (What  are  the  dynamic  load 
characteristics  generated?) 

•  Customization  (Will  baseline  functionality  be 
met?) 


*  Stratification 

6.  Effects  environment 

5.  Mission  environment 
4.  Deployed  Force 
3.  Operationally  ready  capabilities 
2.  Field-able  capabilities 
1.  Equipment  and  bought-in  capabilities 


Landscapes  represent  topological 
characteristics  of  the  system  of  systems 

•  Interoperability  ‘hotspots’  (peaks) 

•  Risks  (gaps  between  peaks) 


-  Software  Engineering  Institute 


Carnegie  Mellon 
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Analysis  for  Synchronization 


Shows  that  the  predominant  mission  awareness  integration  point  is  the 
system  operator  and  the  operator’s  display  console 


11 

10 

9 

8 


q  5 

4 

3 

2 

1 

0 

-1 


Source:  An  Examination  of  a  Structural  Modeling  Risk  Probe  Technique,  Anderson,  Boxer  & 

Brownsword  (2006),  http://www.sei.cmu.edu/publications/documents/06.reports/06sr017.html  NATO  UNCLASSIFIED 
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Analysis  for  Orchestration 


Reveals  areas  of  isolation,  islands  of  high  connectivity, 
and  broad  regions  of  separation 


Source:  An  Examination  of  a  Structural  Modeling  Risk  Probe  Technique,  Anderson,  Boxer  & 
Brownsword  (2006),  http://www.sei.cmu.edu/publications/documents/06.reports/06sr017.html 


NATO  UNCLASSIFIED 


-  Software  Engineering  Institute 
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Analysis  for  Customization 


Source:  An  Examination  of  a  Structural  Modeling  Risk  Probe  Technique,  Anderson,  Boxer  & 

Brownsword  (2006),  http://www.sei.cmu.edu/publications/documents/06.reports/06sr017.html  NATO  UNCLASSIFIED 


-  Software  Engineering  Institute 
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Putting  It  Together 


Where  were  they? 

•  The  organization  was  driven  by  an 
acquisition  focus  for  systems  with  a  pre¬ 
defined  range  of  performance 
requirements. 


System  components 


Where  did  they  need  to  be? 

•  They  needed  to  relate  the  current  state  of 
operational  mission  capability  to  its 
evolving  role  through  its  life. 

What  were  the  gaps? 

•  They  had  no  effective  way  of  managing 
this  cycle  as  a  whole. 


Source:  Managing  the  SoS  Value  Cycle,  Philip  Boxer  (2007) 
http://www.  asymmetricdesign.  com/archives/85 


-  Software  Engineering  Institute 
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Summary 


Systems  of  systems  offer  new  opportunities  and  challenges 

•  Potential  for  greater  range  of  composite  mission  capabilities  orchestrated 
across  systems  of  systems. 

•  Need  for  the  ability  to  continuously  extend  and  adapt  an  operational 
capability  through  its  life  as  a  part  of  a  system  of  systems 

This  presents  a  double  challenge — both  the  institutional  alignment  and  the 
alignment  to  new  and  emerging  forms  of  demand. 

We  can  evaluate  and  characterize  the  gaps  and  risks  by  examining  the 
forms  of  interoperability  possible  within  a  context. 

Providing  methods  to  “work”  the  double  V  as  an  integrated  cycle  can 
provide  the  means  of  mitigating  risks  arising  from  this  dynamic  (re-) 
alignment  through  the  life  of  the  military  operational  capability. 


-  Software  Engineering  Institute 
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For  More  Information 


Philip  Boxer 

pboxer@sei.cmu.edu 

Lisa  Brownsword 

llb@sei.cmu.edu 

Ed  Morris 

eim@sei.cmu.edu 
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Developmental  Test  &  Evaluation 

Policy  Vectors 

Ms.  Darlene  Mosser-Kerner 

T&E  Policy 


Developmental  Test  &  Evaluation 
OUSD(AT&L)/Systems  &  Software  Engineering 


10/24/2007 


1 


DEVELOPMENTAL  T&E 


Outline 


>  Intro  to  OSD  DT&E 

>  DT&E  Mission,  Roles  and  Functions 

>  DT&E  Priorities 
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T&E  -  From  Concept  to  Combat 


Common  Threads  Through  Breached  Programs 

DEVELOPMENTAL  T&E 

•  Nine  key  failures  visible  in  current  Nunn-McCurdy  breaches: 

-  Change  in  doctrine,  driving  quantity  or  mission  changes 

-  Requirements  problems  (unrealistic,  not  stable,  creep,  etc.) 

-  Lack  of  a  robust  baseline 

-  Inadequate  SE  /  T&E,  risk  management,  and  or  FMECA 

-  Inadequate  staffing  /  experience  /  oversight  levels 

-  Poor  reliability 

-  Acquisition  reform 

-  Schedule  /  cost  realism  (concurrency,  estimation,  etc.) 

-  Contract  (warranty,  price  curves,  TSMR,  etc.) 
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T&E  -  From  Concept  to  Combat 


Top  10  Emerging  Systemic  Issues 

DEVELOPMENTAL  T&E 


1.  Management 

•  IPT  roles,  responsibilities,  authority,  poor  communication 

*  Inexperienced  staff,  lack  of  technical  expertise 

2.  Requirements 

*  Creep/stability 

*  Tangible,  measurable,  testable 

3.  Systems  Engineering 

•  Lack  of  a  rigorous  approach,  technical  expertise 

•  Process  compliance 

4.  Staffing 

5.  Reliability 

*  Inadequate  Government  program  office  staff 

*  Ambitious  growth  curves,  unrealistic  requirements 

*  Inadequate  “test  time”  for  statistical  calculations 

6.  Acquisition  Strategy 

•  Competing  budget  priorities,  schedule-driven 

•  Contracting  issues,  poor  technical  assumptions 

7.  Schedule 

*  Realism,  compression 

8.  Test  Planning 

9.  Software 

*  Breadth,  depth,  resources 

*  Architecture,  design/development  discipline 

*  Staffing/skill  levels,  organizational  competency  (process) 

10.  Maintainability/Logistics 

*  Sustainment  costs  not  fully  considered  (short-sighted) 

•  Supportability  considerations  traded 

Major  contributors  to  poor  program  performance 
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T&E  -  From  Concept  to  Combat 


Early  Lifecycle  Planning 

DEVELOPMENTAL  T&E 

•  Early  lifecycle  involvement  of  Systems  Engineering  to: 

-  Inform  evaluation  of  alternatives  with  technical  insights 

-  Ensure  solutions  balance  requirements  with 
technical  feasibility 

-  Ensure  solutions  can  be  validated  and  verified 

-  Use  Modeling  &  Simulation  to  help  refine  warfighter  concept  of 
operations/system  requirements,  evaluate  design  alternatives,  and 
identify  potential  technology/human  interface  constraints 

•  Appropriate  resourcing  (personnel/funding)  required 

•  Include  in  requirements,  specifications,  and  contracts 


MSA 


Sustainment  must  be  included  up  front  and  early 


T&E  in  Support  of  Systems  Engineering 

DEVELOPMENTAL  T&E 


^steins  Engineer/^ 


\ 


User  Require 
rfients  &  Concept 
of  Operations 

r 


involvement 


System 
Requirements 
&  Architecture 


System 
Demonstration 
&  Validation 

I 


Component 

Design 


Component 
Integration 
&  Test 


Procure,  Fabricate, 
&  Assemble 
Parts 


DT&E 

Verification 
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T&E  -  From  Concept  to  Combat 


DEVELOPMENTAL  T&E 


Where  am  I  in  OSD? 
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T&E  -  From  Concept  to  Combat 


DEVELOPMENTAL  T&E 


Our  Mission 


•  Lead  office  within  the  DoD  for  all  matters  pertaining  to 
developmental  test  and  evaluation 

-  Develops  OSD  policy  concerning  DT&E 

-  OSD  advocate  for  testers  concerning  DT&E 

-  Responsible  for  education/training  of  the  T&E  acquisition 
workforce 

•  Office  of  primary  responsibility  for  DoD  Energy  acquisition  policy 

-  Emerging  area  of  emphasis  on  new  weapon  system 
development 

•  Lead  office  for  acquisition  M&S  and  System  Safety 
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T&E  -  From  Concept  to  Combat 


DEVELOPMENTAL  T&E 


The  Direction  We  are  Heading 


-  Revitalizing  DT&E 

•  Department  initiative  to  place  more  emphasis  on  government 
DT&E  during  system  development 

-  Integrated  Test  policy 

•  Standardizing  definitions  and  execution  guidance  throughout 
the  Services  and  OSD 

-  Testing  in  a  Joint  Environment 

•  Several  ongoing  initiatives  (JTEM,  L-V-C,  DIVIO,  etc) 
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T&E  -  From  Concept  to  Combat 


DEVELOPMENTAL  T&E 


The  Need  to  Revitalize  DT&E 


-  Too  many  acquisition  programs  not  operationally  effective  or 
suitable 

•  Several  reasons  postulated  as  cause  -  reduction  in 
governmental  DT&E? 

-  Policy  has  languished  concerning  governmental  involvement 
during  system  development 

-  DT  data  typically  not  relevant  to  the  evaluation  of  a  system’s 
operational  readiness 

•  Scope  is  concentrated  on  more  technical  parameters 

-  DT  focused  on  single  system  development 

•  Needs  wider  emphasis  on  system  of  system  and/or  system 
employment  in  a  joint  context 
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T&E  -  From  Concept  to  Combat 


A  New  Vector  for  DT&E 

DEVELOPMENTAL  T&E 

•  Support  Faster  Fielding  of  Improved  Capabilities 

•  Reduce  Risk  of  Immature  Technology  in  Systems  Development 

•  Revitalize  T&E  Workforce  Education 

•  Promote  Joint  T&E  in  Live-Virtual-Constructive  Environments 

•  Provide  Effective  Acquisition  Policy  and  Practices  for  DT&E 
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T&E  -  From  Concept  to  Combat 


Support  Faster  Fielding  of  Improved  Capabilities 

DEVELOPMENTAL  T&E 

Objective:  Develop  T&E  policy,  practices,  and  procedures  to  support 
Departmental  efforts  in  shortening  the  time  to  field  capabilities 
>  Issues: 

>  Not  pass-fail;  but  based  on  capabilities  and  limitations 

>  Integrate  T&E  strategy  -  CT,  DT,  OT 

>  Incorporate  operational  context  in  DT 

>  Collect  once,  and  use  data  often  -  Integrated  Testing 

>  Ensure  testable  requirements  are  in  EoA  /  CD 

>  Ensure  T&E  requirements  are  in  SOWs  and  RFPs 

>  Ensure  T&E  documents  consistent  with  and  support: 

>  Systems  Engineering  Plan  (SEP) 

>  Acquisition  Strategy  (AS) 

>  Capability  Documents  (ICD,  CDD,  and  CPD) 
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T&E  -  From  Concept  to  Combat 


Reduce  Risk  of  Immature  Technology  in 

Systems  Development 

DEVELOPMENTAL  T&E  I 

Objective: 

•  Add  Technology  Maturity  focus  into  the  Systems  Engineering  and 
DT&E  processes  to: 

-  Reduce  technical,  cost,  and  schedule  risk 

-  Increase  the  rigor  of  SE 

-  Plan  for  alternatives  in  the  event  of  TM  difficulty 

-  Verify  TRLs  during  DT&E 
Scope 

•  Leverage  existing  acquisition  review  structure 

MSA  MSB  CDR  MS  C 

•  Use  existing  DDR&E  Technology 
Readiness  Assessment  (TRA) 
methodology 

(/j 

CO 
-C 
Q. 

E 

CD 
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T&E  -  From  Concept  to  Combat 


time 


Reduce  Risk  of  Immature  Technology  in 

Systems  Development 

DEVELOPMENTAL  T&E  y  r 

Issues: 

•  Studies  find  that  immature  technology  is  a  primary  source  of  cost 
and  schedule  risk 

-  GAO  -  DAPA 

-  QDR  --  SSE/AS  Program  Support  Reviews 

•  “Programs  that  started  development  with  immature  technologies 
experienced  an  average  acquisition  unit  cost  increase  of  nearly  21 
percent”  (GAO-05-301,  March  2005) 

•  FY06,  PL  109-163,  Section  801  requires  USD(AT&L)  certification, 
before  Milestone  B,  that  “the  technology  in  the  program  has  been 
demonstrated  in  a  relevant  environment” 

-  Above  wording  equates  to  Technology  Readiness  Level  (TRL)  6 
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T&E  -  From  Concept  to  Combat 


Technology  vs.  Technical  Maturity 

DEVELOPMENTAL  T&E 


15 


10/24/2007 


T&E  -  From  Concept  to  Combat 


Increased  TM  emphasis  in  OSD  Oversight 

DEVELOPMENTAL  T&E 

•  Program  Support  Review  (PSR) 

-  ID  Critical  Technology  components/sub-systems? 

-  Current  TRLs  known? 

-  ID  Mature  alternative  components/sub-systems? 

-  TRL  monitoring,  Alternative  decision  date? 

•  Assessment  of  Operational  Test  Readiness  (AOTR) 

-  TM  verification  results 

-  DT&E  performance  results 

-  IOT&E  predictive  analysis/M&S 
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T&E  -  From  Concept  to  Combat 


Revitalize  T&E  Acquisition  Workforce 

Education 

DEVELOPMENTAL  T&E 

Objective:  Ensure  the  T&E  acquisition  workforce  is  of  sufficient  size 
and  adequately  trained  to  perform  the  T&E  tasks  required  in  today’s 
and  tomorrow’s  product/system  acquisition  process 

Issues: 

>  Continue  to  ensure  current  &  relevant  education,  experience,  training 
requirements 

>  Track  new  DAU  course  releases 

>  Identify  the  T&E  education  requirements  for  SoS  and  FoS 

>  Champion  the  development  of  new  CLMs  -  such  as  “M&S  for  T&E” 
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T&E  -  From  Concept  to  Combat 


DT&E  Acquisition  Education  &  Training 

DEVELOPMENTAL  T&E 


Common  acquisition 
foundation 
knowledge  and  skills 


Career  Field 
foundation 
knowledge  and  skills 


“Plus”  or  job  competency 
point-of-need 
training  (frequently  CLMs) 


Web-link.  https://acc.dau.mil/Core  Plus 

10/24/2007  T&E  -  From  Concept  to  Combat 


10/24/2007 


Promote  Joint  T&E  in  Live-Virtual- 

Constructive  Environments 

DEVELOPMENTAL  T&E 

Objective:  Define  the  role  of  DT&E  in  the  joint  T&E  arena  and  partner 
with  DOT&E,  Joint  Staff,  and  Components  in  defining  and 
developing  the  necessary  polices,  practices,  and  procedures  for  the 
conduct  of  efficient  and  effective  joint  T&E 

Issues: 

>  Establishing  L-V-C  standards 

>  Defining  LVC  environment  functional  requirements 

>  Identify  capabilities  &  limitations  of  LVC  architectures 

>  Map  capabilities  &  limitations  to  requirements 

>  Compare  middleware,  business  models,  standards  management, 
alternatives 

>  Create  roadmap,  and  socialize  it  widely 

>  Define  business  processes 

>  Establish  a  Transition  Plan  to  include:  who  pays,  legacy  implementation, 
etc. 
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T&E  -  From  Concept  to  Combat 


Testing  in  a  Joint  Mission  Environment 

DEVELOPMENTAL  T&E 


Upcoming  changes  in  OSD  policy 
will  likely: 

>  Require  testing  in  a  joint 
environment  for  capabilities- 
based  acquisitions 

>  Establish  governance  on  the 
use  of  the  joint  mission 
infrastructure 

>  Enable  smaller  programs  to 
participate  and  contribute  to  the 
joint  environment 


>  Increase  demonstration  venues 
for  systems  earlier  in 
acquisition  cycle 
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T&E  -  From  Concept  to  Combat 


Provide  Effective  Acquisition  Policy  and 

DEVELOPMENTAL  T&E 

Objective:  Develop  and  socialize  the  necessary  changes  to  DT&E 
policy,  practices,  and  procedures  to  support  the  overall  AT&L 
acquisition  lifecycle  management  framework  and  process 

Issues: 

>  More  involvement  in  the  Evaluation  of  Alternatives  and  Concept  Decision 

>  Involvement  in  Capabilities  design  &  SoS  T&E 

>  Develop  a  format  for  T&E  Strategy  (TES) 

>  Reinforce  Integrated  T&E  approach  in  TES  /  TEMP 

>  Enforce  linkage  of  T&E  and  SE  planning  documents 

>  Incorporate  Industry  best  practices 

>  Incorporate  DT&E  standards  for: 

>  Early  involvement  (requirements  definition  in  Concept  Refinement) 

>  Increased  operational  perspective,  operator  involvement 

>  System  sustainment  issues 

>  Open  processes  and  data  availability 

>  M&S  part  of  T&E  strategy;  live  test  data  used  to  improve  M&S 

21  10/24/2007  T&E  -  From  Concept  to  Combat 


Practices  for  DT&E 


2007 NDIA  SE/DT&E  Committee  Focus 

DEVELOPMENTAL  T&E 

•  Three  Focus  Teams: 

-  Earlier  contractor  and  tester  involvement 

-  Integrated  DT/OT  and  DT  operational  relevance  (combined) 

-  Suitability 

•  Recommend  policy  changes 

-  Input  to  FY2008  DoD  5000  update 
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T&E  -  From  Concept  to  Combat 


DEVELOPMENTAL  T&E 


Summary 


New  Approaches  to  Acquisition: 

•  Emphasis  on  evolutionary  acquisition 

•  Joint  capabilities  focus 

•  Net  Centricity 

•  Sy  ste  m  -of-Sy  ste  m  s 

•  Testing  in  a  joint  mission  environment 

Need  a  revitalized  DT&E  capability  to  be  a  productive  team  member 
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DoD  Systemic  Root  Cause  Analysis 


Dave  Castellano 

Deputy  Director,  Assessments  and  Support 

Laura  M.  Dwinnell 

Systemic  Analysis  Team  Leader 


Systems  &  Software  Engineering 
Office  of  the  Deputy  Under  Secretary  of  Defense 
for  Acquisition  and  Technology 

23  October  2007 


VERSION  1.01 


Systems  and  Software  Engineering...  What 

are  we  all  about? 


Acquisition  Program  Excellence  through 
sound  systems  and  software  engineering ... 


•  Help  shape  portfolio  solutions  and  promote  early  corporate  planning 


•  Promote  the  application  of  sound  systems  and  software  engineering , 
developmental  test  and  evaluation ,  and  related  technical  disciplines 
across  the  Department's  acquisition  community  and  programs 


•  Raise  awareness  of  the  importance  of  effective  systems  and 
software  engineering,  and  drive  the  state-of-the-practice  into 
program  planning  and  execution 


•  Establish  policy,  guidance,  best  practices,  education,  and  training  in 
collaboration  with  academia,  industry,  and  government  communities 


•  Provide  technical  insight  to  the  leadership  to  support  effective  and 
efficient  decision  making 

Based  on  USD(AT&L)  2004  Imperative ... 

“Provide  context  within  which  I  can  make  decisions  about  individual  programs.” 
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Providing  Value  Added  Oversight 

&  Support 


Tactical,  Program  and  Portfolio  Management 


“a” 


PEOs&PMs... 

•  PSR 

•  AOTR 

•  SEP 

•  TEMP 

•  DAES 

Improved  Program 
Execution  thru... 
Program  Unique 
Recommendations 


AS 

Results 
Achieved  thru 

Open  Communication/Debate 
Insight  &  Information  Sharing 
•  Understanding  of 
Consequences 
•  Data  Driven,  Fact-based 
Information 
Synthesis 


Acquisition  Leadership 

Improved  Acquisition  Decision 
Making  thru... 

•  Greater  Program  Transparency 

•  Acquisition  Insight 


Strategic  Management 


DoD  Acquisition  Community 

Improved  Acquisition 
Support  to  Warfighter 


•  Systemic  Issues  &  Risks 

•  Systemic  Strengths  &  Indicators 

Improved  Acquisition 

Support  to  Warfighter 

Recommendations 

Policy/Guidance 
Education  &  Training 


Best  Practices 

Other  Processes  (JCIDS,  etc) 


Oversight  (DABS/ITAB) 
Execution  (staffing) 
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Value  Added  Oversight 


Systemic  Analysis:  Data  Mode! 


Steps  la,  1b,  2-4 

Underway.. 


Tactical,  Program  and 
Portfolio  Mgt 


Program 

Review 

Findings 


Program 
Causes- 
Effects  & 
Root  Causes- 


Program 

Unique 

Solutions 


SEP 
Findings  IJ-1- 


&E 
Findings  IH 


Strategic  Management 


Other 
Findings  IH 


TBD 


Systemic 

Issues 


Systemic 

Root 

Causes 


Corrective 

Actions 


DoD  Acquisition 
Community 


Policy/Guidance 
Education  &  Training 
Best  Practices 


Other  Processes  (JCIDS, 
Oversight  (DABS/ITAB) 
Execution  (staffing) _ 


etc) 
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Top  10  Emerging  Systemic  issues 

(from  52  Program  Reviews  since  Mar  04) 


1.  Management 

•  IPT  roles,  responsibilities,  authority,  poor  communication 

•  Inexperienced  staff,  lack  of  technical  expertise 

2.  Requirements 

•  Creep/stability 

•  Tangible,  measurable,  testable 

3.  Systems  Engineering 

•  Lack  of  a  rigorous  approach,  technical  expertise 

•  Process  compliance 

4.  Staffing 

•  Inadequate  Government  program  office  staff 

5.  Reliability 

•  Ambitious  growth  curves,  unrealistic  requirements 

•  Inadequate  “test  time”  for  statistical  calculations 

6.  Acquisition  Strategy 

•  Competing  budget  priorities,  schedule-driven 

•  Contracting  issues,  poor  technical  assumptions 

7.  Schedule 

•  Realism,  compression 

8.  Test  Planning 

•  Breadth,  depth,  resources 

9.  Software 

•  Architecture,  design/development  discipline 

•  Staffing/skill  levels,  organizational  competency  (process) 

10.  Maintainability/Logistics 

•  Sustainment  costs  not  fully  considered  (short-sighted) 

•  Supportability  considerations  traded 

Major  contributors  to  poor  program  performance 
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Observations  Since  Last  Year 


•  Programs  fail  because  we  don’t... 

-  Start  them  right 

-  Manage  them  right 


...  We  Don't  Start  Them  Right 


•  Requirements  creep/stability  -  not  tangible,  measurable,  testable, 
defined 


•  Acquisition  strategies  based  on  poor  technical  assumptions, 
competing  budget  prioritities,  and  unrealistic  expectations 

•  Budget  not  properly  phased 

•  Lack  of  rigorous  systems  engineering  approach 

•  Schedule  realism  -  success  oriented,  concurrent,  poor  estimation 
and/or  planning 

•  Inadequate  test  planning  -  breadth,  depth,  resources 

•  Optimistic/realistic  reliability  growth  -  not  a  priority  during 
development 

•  Inadequate  software  architectures,  design/development  discipline, 
and  organizational  competencies 

•  Sustainment/life-cycle  costs  not  fully  considered  (short-sighted) 
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...  We  Don't  Manage  Them  Right 


•  Insufficient  trade  space  -  resources,  schedule, 
performance,  requirements 

•  Inadequate  IMP,  IMS,  EVMS 

•  Insufficient  risk  management 

•  Concurrent  test  program 

•  Inadequate  government  PMO  staff 

•  Inexperienced  and/or  limited  staffing 

•  Poorly  defined  IPT  roles,  responsibilities  and 
authority 

•  Poor  communications 
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Root  Cause  Effects  Model 


[ 


Solution  Set 


Systemic 
Root  Cause 


Technical  Process 
Management  Process 
Acquisition  Practices 
Requirements  Process 
Competing  Priorities 
Staff 

Communication 
Program  Realism 


Contract  Structure  & 
Execution 


Etc 


Systemic  Issues 


Symptoms 


Who’s  Affected 


Management 

Requirements 

Systems 

Engineering 

Staffing 

Acquisition  Strategy 

Schedule 

Test  Planning 

Software 

Maintainability  & 
Logistics 


execution  risk 
Potential  schedule  and  cost 
breach 

Shared  engineering 
functions  not  given  proper 
attention 
Rework 

Insufficient  system 
performance  information  to 
make  informed  milestone 
decision 

Potential  for  lower  readiness 
levels  and  higher  maintainer 
workload 
Etc... 
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Systemic  Analysis  Milestones 


Develop  & 
pilot  root 
cause  terms 

III 

Conduct 
SRCA 
Workshop 
(Part  1) 

l| 

Apply  Root  Cause 
Structure  to 
program  findings 

III 

Analyze 

preliminary 

results 

1 

Conduct 

SRCA 
Workshop 
(Part  II) 

1 

Oct  06 

- n - 

Jan  07 

Feb- Jul  07 

Aug  07 

Sep  07 

*  Categorized  root  cause  textual  descriptions 

*  Terminology  developed  by  small  team,  limited 

*  Pilot  effort  proved  that  terms  lacked  proper 
structure  and  definition 


*  Redefined  Root  Cause  Type:  3  Tier 

*  Terminology  developed  by  workshop 
participants  representing  DoD  and  Industry 

*  RCT  structure  informally  tested  on  4  programs 
from  different  domain  areas 


Pilot  RCT  on  program  reviews:  past  effort  and 
go-forward 

Definitions  enhanced,  terminology  revised 
Analysis  of  trends  and  applicability; 


Validate  pilot  on  root  cause  method/structure 
Formulate  systemic  root  cause 
recommendations 

Feedback  on  SA  model  and  root  cause 
methodology 


Coming  Up: 

Oct  07:  Present  results  to  SE  community  (NDIA-SE  Conference) 
Nov07:  Present  results  to  acquisition  community  (PEO  SYSCOM) 
Dec  07:  Formalize  and  standardize  methodology 
Mar  08:  Incorporate  other  data  sources  (SEP,  Triage,  etc) 


Expand  analysis  to  complete  data  set 
Establish  NDIA  Working  Group  on  SRCA 
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Root  Cause  Types 
Recap  of  Part  /  Results 


.# 


ii 

\ 


\ 


\ 


Root  Cause  Types  needed  to  categorize  and  discuss 
root  causes 

Root  Cause  Type  structure  defined 

-  Tier  1 :  Root  Cause 

»  Textual  description;  documented  by  PSR  team 
»  Perceived  projrarajoot  ea^ 

-  Tier  2>  Bvstemic  Root  Cause 

^  *  *  From  pre-defined  list;  assigned  by  PSR  team 


X 


'X. 


"■•X. 


» 


"'"X. 


X. 


X 


Can  be  "A"  or  "a".  Conditions  that  are  outside  the  PMO  below  the 
Defense/Service  Acquisition  Executive  level.  This  would  include  lateral  \ 
activities,  such  as  Service  staff  functions  (OPNAV,  Air  Staff,  etc.)  and  the  \ 
system  commands. 

Tier  3:  Core  Root  Cause  a 

»  From  pre-defined  list;  assigned  by  PSR  team 

»  At  the  "A“  level.  Something  at  the  DAE  level  (3  Star  level  and  above} - 
-  ^  Issues  resolved  through  DAE  coordination  with  Congress,  DoD W  ^  ^ 

"  Services,  Industry,  etc, 


Root  Cause  Analysis  is  Crux  of  Systemic  Solutions 
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Root  Cause  Type  Structure 


♦  Systemic  Root  Cause  (Tier  2) 


•  Core  Root  Cause  (Tier  3) 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 
11. 
12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 
21. 
22. 

23. 

24. 

25. 
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Ineffective  communication 

Competing  priorities 

CONOPs  change 

Definition  of  enterprise 

Engagement  of  supply  base  in  SE  process 

Expectations  not  defined 

Inadequate  baseline  management 

Inadequate  contract  structure  and  execution 

Inadequate  cost  metrics  e.g.  EVMS 

Lack  of  accountability 

Lack  of  capital  investment 

Lack  of  enterprise  wide  perspective 

Lack  of  appropriate  staff 

Lack  of  trade  space/constraints 

Lack  of  trust  and  willingness  to  share  information 

Obfuscating  bad  news 

Ineffective  organization 

Poorly  defined  roles/responsibilities 

Process  -  Management 

Process  -  Production 

Process  -  Requirements 

Process  -  Technical 

Program  realism 

Responsibility  w/o  authority 

Poor  Acquisition  Practices 


1.  Acq  Reform:  Loss  of  govt,  capital  investment 

2.  Acq  Reform:  Loss  of  MS  A  requirement 

3.  Acq  Reform:  Transferred  Authority 

4.  Enabling  infrastructure 

5.  Budget  POM  process  (PBBE) 

6.  Culture 

7.  Rotations  /  continuity 

8.  Inadequate  JCIDS  process 

9.  Pool  of  clearable  skilled  people 

10.  External  influences 

1 1 .  Poor  business  practices 
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SADB  Features 


H  Systemic  Analysis  Database  (SADB)  -  Microsoft  Internet  Explorer 


File  Edit  View  Favorites  Tools  Help 
Q  Back  -  @  @  ifj]  |  f-  ' Search 


Writing  Pad  ^ 


42>|  ^ 


|  Address  W  https :  //atstdvp  1 ,  pica ,  army ,  mil/sadb/MainMenu .  jsp 


v]  0  Go  \  Links  \  Googk  |[CK  >y  3  Settings  t  3  McAfee  Site  Advisor  T  |i 


SYSTEMIC  ANALYSIS  DATABASE 


Relational 
Web-enabled 
Excel  output 
Embedded  charts 
Search  wizards 
Data  query 

Data  quality  reporting 


Sponsored  By: 
ODUSD  { A Er T }  SSE 
Assessments  and  Support 

FOUO  (Pie-Decisional) 


Database  Developed  By: 
RDECOM  -  ARDEC  PI  CAT  IN  NY,  MJ 
Fire  Control  Systems  £t  Technology 
Automated  Test  Systems  Division 


Done 


Si  «  Internet 
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Systemic  Root  Cause  Analysis 

Preliminary  Results 


•  Analysis  performed  on  44  program  reviews 

•  SRCA  applied  to  negative  findings:  ~  48%  of 
total  set,  -1500  findings 

•  Trends  shown  by: 

-  (1)  Systemic  Root  Cause  (SRC) 

-  (2)  DAPS  areas  related  to  leading  SRC 

-  (3)  Core  Root  Cause  (CRC) 

-  (4)  SRCs  as  related  to: 


»  CRC  =  Poor  Business  Practice 
»  CRC  =  Culture 


See  Next  5  Slides  for  Results... 
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No.  of  Programs 


Categorization  by  Systemic  Root  Cause 


Number  of  Programs  that  Experienced  the/SRC  (P< 


•Occurred  on  50%  or 
more  of  the 
programs 
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Systemic  Root  Cause 


Note:  Chart  shows  13  of  26  SRCs 
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Systemic 
Root  Cause 


•*. 
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Systemic  Root  Cause:  Technical  Process 


For  SRC  =  Technical  Process:  Number  of  Programs  that  Experience  Issues  by  DAPS 


Program  Issues 
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Root  Cause  Themes 


•  Aggressive,  success-oriented,  highly  concurrent  test  schedule 

•  Reliability  not  progressing  as  planned  or  has  failed  to  achieve  requirements 

•  Software  reuse  was  significantly  less  than  planned  or  expected 

•  Testing  and  verification  approach  are  inadequate 

•  Program  has  inadequate  systems  engineering  process 
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Categorization  by  Core  Root  Cause 
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Relationship  between  CRC  and  SRC 


For  CRC  =  Poor  Business  Practices:  Number  of  Programs  that 

Experienced  SRC 
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Relationship  between  CRC  and  SRC 
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SRCA  Workshop  Participants 

(Part  II)  25-26 Sep  07 


•  Approximately  33  participants  representing  government  and 
industry 

•  Non-OSD  participants  included... 


-  Government 

»  Col  Horejsi,  US  Air  Force  (PEO) 

»  Mr.  George  Mooney,  USAF  CSE 
»  Ms.  Kathy  Lundeen,  DCMA 
»  Mr.  John  Snoderly,  DAU 

-  Industry 

»  Mr.  Bob  Rassa,  NDI A/Raytheon 
»  Mr.  Brian  Wells,  Raytheon 
»  Mr.  Rick  Neupert  &  Mr.  Jamie  Burgess,  Boeing 
»  Mr.  Stephen  Henry,  Northrop  Grumman 
»  Mr.  Per  Kroll,  IBM 
»  Mr.  Paul  Robitaille,  Lockheed  Martin 
»  Dr.  Dinesh  Verma,  Stevens  Institute  of  Technology 
»  Mr.  Dan  Ingold,  University  of  Southern  California 


Raytheon 


Institute  ofTechnology 


NORTHROP  GRUMMAN 


lllll 


LOCKHEED  MARTIN 


Defense  Acquisition  University 
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SRCA  Workshop  (Part  //  )  Objective 


•  Primary  SRCA  Workshop  II  objective: 


-  Formulate  systemic  root  cause  recommendations 

•  Participants  focused  on  manageable  subset  of  analysis 
results 


-  2  CRC  areas  and  their  top  4-5  SRCs 
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Root  Cause  Model 

(e.g.,  Poor  Business  Practices) 


Source 


Systemic  Root  Cause 
(Top  4) 


Core  Root 
Cause 


Solution  Set 


Management 

Process 


Technical 

Process 


Contract 
Structure  & 
Execution 


Requirements 


Policy/ 

Guidance 


Education 
&  Training 


Best 

Practices 


Governance 


\ 


tions  Must  Address  Root 


NDIA-SRCA  vl.01 


Page  23  of  32 


RECOMMENDATIONS 


Initial  Thoughts  on  Systemic  i mprovement... 


©  Communications 


+  Organization 


Acquisition  Practices  (Planning) 


Systemic  Root 
Cause  Analysis 


Acquisition  Practices 
(Incremental  Toll  Gates  &  © 

Reviews) 


©(Technical  Process  M  Recommendations 


-  >  Requirements  Process  © 


©■ 


Management  Process 
(Incentives  +/-) 


Management  Process 
(IMPs/iMS/WBS) 


Management  Process 
^  (Execution) 


Best 

Practices 
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SRCA  Workshop  Part  //  -  Results 


Over  50 

recommendations 

-  Varied  level  of  detail 

-  Directed  at  variety  of 
sources 

»  Acquirer  & 
Developer 

»  PM,  PEO,  Comp. 
Rep.,  Acq.  Exec 

»  Senior 
Management  to 
Systems  Engineer 
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•  Develop  Action  Plan 

-  Prioritize  the  emerging  recommendations 

-  Assign  stakeholders 

-  Establish  timelines 

•  Complete  analysis  on  remaining  CRC  areas 

•  Formalize  NDIA  Working  Group  to  continue 
recommendation  development  on  CRC  analysis 
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Questions/ Discussion 


Contact  Information: 


Dave  Castellano  Laura  Dwinnell 

ODUSD(A&T)  Systems  &  Software  Engineering  SSE/AS  Support 

Deputy  Director,  Assessments  and  Support  Systemic  Analysis  Team  Lead 

David.Castellano@osd.mil  LDwinnell@fasi.com 


NDIA-SRCA  vl. 01 


Page  27  of  32 


Systemic  Root  Cause  Analysis 
Industry  Panel  Discussion 

Panel  Moderator:  Mr.  Bob  Rassa 
Dave  Castellano 

Deputy  Director,  Assessments  and  Support 

Laura  M.  Dwinnell 

Systemic  Analysis  Team  Leader 

Systems  &  Software  Engineering 
Office  of  the  Deputy  Under  Secretary  of  Defense 
for  Acquisition  and  Technology 


23  October  2007 


Industry  Pane, /  Members 


Mr.  Stephen  Henry 

-  Northrop  Grumman:  Principal  Engineer 

Mr.  Brian  Wells 

-  Raytheon:  Chief  Systems  Engineer 

Mr.  Per  Kroll 

-  IBM:  Manager  -  Methods  IBM  Rational 

Mr.  Paul  Robitaille 


NORTHROP  GRUMMAN 


Raytheon 


-  Lockheed  Martin:  Director  of  Systems  Engineering 

Lockheed  Martin  Corporate  Headquarters;  L0Ckheed  martin  / 

President,  INCOSE  X 


Mr.  James  Burgess 

-  Boeing:  Systems  Engineering  Senior  Manager, 
Leader  of  the  Boeing  Systems  Engineering  Best 
Practices  Initiative  Boeing  Integrated  Defense 
Systems 
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Results  -  SRCA  Workshop  Part  1 1 


5  “Heavy  Hitter”  recommendations  include: 


1 .  Increase  or  improve  competition  -  down  select  at 
SRR/PDR/CDR 

2.  Provide  mechanisms  for  better  performance  & 
Implement  consequences  for  non-performance 

»  Increase  use  of  toll  gate  reviews  with  off-ramps  and  specific 
guidance/requirements 

3.  Ensure  better  definition  and  verification  of 
requirements.  E.g.  use  meta-language,  SE-based 
modeling,  etc. 

4.  Require  more  close  coupling  of  the  IMPs/IMS/WBS 

5.  Increase  acquisition  workforce  and  expertise 

»  Use  “green  teams”  to  augment  needed  acquisition  expertise 
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When  is  Extended  Competition  Cost 

Effective? 


Program  Complexity  & 

SW  Growth 

ATP 

SRR 

PDR 

CDR 

Medium  High  Complexity 

Holchin  Level  7* 

188% 

144% 

122% 

111% 

Down  Select  Cost 
Savings  Medium  High 

34% 

31% 

-3% 

Medium  Low  Complexity 

Holchin  Level  3* 

144% 

122% 

111% 

106% 

Down  Select  Cost 
Savings  Medium  Low 

12% 

-2% 

*  SW  Growth  Based  on  Holchin  Growth  Curve  Average  Growth 
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Questions/ Discussion 


Contact  Information: 


Dave  Castellano  Laura  Dwinnell 

ODUSD(A&T)  Systems  &  Software  Engineering  SSE/AS  Support 

Deputy  Director,  Assessments  and  Support  Systemic  Analysis  Team  Lead 

David.Castellano@osd.mil  LDwinnell@fasi.com 
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Systems  Engineering  in  the  Cognitive 
and  Social  Domains  of  NetCentric 

Operations 


Abe  Meilich,  Ph.D.,  C.C.P. 
Lockheed  Martin  Corporation 
Simulation,  Training,  and  Support 
abraham.w.meilich@lmco.com 


National  Defense  Industrial  Association 
10th  Annual  Systems  Engineering  Conference 

San  D/ego,  CA  _ 

October  24,  2007 


NDIA  -  Dr.  Abe  Meilich 


So,  Where  Do  We  Go  From 


“Becoming  net-centric  is  not  about  replacing  the  warfighter  wjth 
technology.  We  wIIl  Tor  example,  ~siili  needboots  on  the  ground.  Net¬ 
centric  operations  will  allow  humans  to  leverage  information  to  better 
deal  with  unanticipated  challenges ,  needs,  partners,  and 
circumstances.  ” 

“ 'Enabling  Technologies  for  Net-Centricity  -  Information  on  Demand ”, 
John  Grimes  (Department  of  Defense  Chief  Information  Officer), 
CrossTalk,  July  2007 


NDIA  -  Dr.  Abe  Meilich 


2 

10/24/07 


Topics 


•  NCO  and  pumans  in  the  Loop  (HITL) 

•  The  Human  As  a  Key  Consideration  in  SoS 

•  Observations  on  Human  Systems  Integration  (HSI) 

•  Implication  from  DOTMPLF  on  System  Engineering 

•  Applying 

•  Observation  from  the  Perspective  of  Operations  Analysts 

•  Considerations  for  Systems  Engineering 

•  Observations  on  Experimentation 

•  Summary  Comments  on  Engineering  in  the  Cognitive  and  Social 
Domains 


NDIA  -  Dr.  Abe  Meilich 
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Background 


•  Net  Centric  Operations  implies 

-  Leverage  SoS  in  new  and  different  ways 

-  “Potential”  for  decision  makers  and  operators  to 

•  Unprecedented  access  to  information  and  assets  over  the  network 

•  More  effective  and  efficient  human  “networking” 

•  Does  Faster  information  yield  smarter  decisions? 

•  Leveraging  ubiquitous  and  COI-specific  services  with  pre-engineered 
interoperability  in  new  and  different  ways 

-  For  the  human  this  will  require  semantic  interoperability;  among  many 
other  interoperability  attributes 

•  Bottom  Line  Goal  »  “The  DoD’s  NCE  is  a  framework  for  human  and 
technical  connectivity  and  interoperability  that  allows  DoD  users  and 
mission  partners  to  share  and  protect  information  and  to  make  informed 
decisions.”  * 

•  Engineering  for  the  Human  -In-The-Loop  (HITL)  in  a  SoS 

-  Evolving  area  of  research 

•  But,  implementation  of  SoS  ahead  of  application  of  theory  of  networking 
over  the  SoS 

-  e.g.,  in  NCO  -  more  sources,  more  deconfliction,  more  sensemaking  required 

•  Solution:  Historically,  engineer  through  experimentation 

•  Dilemma:  But  what  do  we  need  to  measure  and  for  what  purpose? 


*  “DOD  NetCentric  Services  Strategy:  Strategy  for  a  Net-Centric,  Service  Oriented  DoD  Enterprise”, 
DOD  CIO,  Washington,  DC,  May  4,  2007. 
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NCO  Value  Chain 


Information  Domain 


Common 
Operational 
Picture 


Collaboration 


The  ffuroan  As a  Key  Consideration  in  SoS 


•  Collaborative  decision-making  and  shared  situation  awareness  amongst 
the  human  operators.  -  key  to  SoS* 

•  More  research  in  human  system  interaction  and  decision-making  is 
required  to  understand  better  how  to  integrate  these  elements  into  an 
effective  system-of-systems  architecture. 

-  Systems  Engineering:  human-machine  interface  (HMI)  was  the  focus  for  a 
single  system;  human-to-system  and  human-to-human  interaction  is  the 
focus  for  SoS 

•  Think  of  the  OODA**  Loop: 

-  How  do  we  engineer  a  capability  to  improve  the  OODA  without  considering  the 
HITL? 

-  Clearly,  the  cognitive  processes  that  allow  the  human  to  perceive  and  decide 
are  at  the  center  of  the  human  dimensions  of  war 

-  Remember:  Capabilities  are  written  in  terms  of  what  the  human  needs  to 
accomplish  ,  not  what  the  machine  must  do 


*  US  AFSAB,  Executive  Summary  and  Annotated  Brief,  SAB-TR-05-04,  July  2005 
**  OODA:  observe,  orient,  decide,  act 
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The  Human  As  a  Key  Consideration  in  SoS 


•  Including  human  system  interaction  as  a  part  of  System-of-Systems 
Engineering  continues  to  be  an  important  design  parameter 

•  How  do  we  design  around  the  ad  hoc,  transaction-oriented  situations 
that  are  common  in  military  situations? 

•  Many  designs  assume  a  fixed  scenario  as  the  basis  for  design  ala 
DODAF 

-  DODAF  not  sufficient  for  evaluation  of  “human  behavior” 
utilizing  information  systems  in  a  SoS  environment 

•  Systems  are  designed  at  the  individual-to-system  interface;  in  a 
SoS,  NetCentric  environment  we  should  be  designing  to  account 
for  how  individuals,  crews,  teams,  units,  or  organizations  interact 
with  the  systems,  and  in  a  context  of  a  SoS 
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Observations  on  HSI 


•  SOSE  must  be  done  in  a  context  of  users,  operators,  maintainers, 
and  support  personnel  in  operational  environments  which  may 
complementary  as  well  as  have  conflicting  needs 

-  Total  ownership  costs  includes  all  of  the  above 

-  Many  do  not  want  to  consider  all  the  above,  since  it  may  drive  up  total 
ownership  costs  of  a  proposed  or  envisioned  SOS 

-  Above  and  beyond  the  technology  implementation  of  a  single  system 
we  have  the  effects  of  security  and  safety  (both  driven  by  human 
behavior)  multiplied  -  but,  that  is  the  value  that  SOSE  brings  to  the 
table 

•  HSI  ensures  that  human-centered  domains  are  integrated 
throughout  system  design,  development,  manufacturing, 
operation,  sustainment,  and  disposal 

•  HSI  seeks  to  treat  humans  equally  with  other  system  elements, 
such  as  hardware  and  software,  in  system  design 


*INCOSE  Handbook,  2007 
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Observations  on  HSI 


•  -‘Effective  front-end  analyses  start  with  a  thorough 
understanding  of  the  mission  of  the  new  system, 
successes  or  problems  with  any  predecessor  systems, 
systems  with  which  the  proposed  system  must 

^interface,  and  the  Knowledge,  skills,  abilities,  and 
training  associated  with  the  people  who  are  likely  to 
interact  with  the  proposed  system.”  * 

-  In  addition,  we  must  multiply  the  complexity  of  this  analysis 
in  a  SoS 

•  There  is  controversy  amongst  the  HSI  discipline 
practitioners  as  to  how  HSI  works  in  conjunction  with 
SE  (i.e.,  is  it  a  separate  discipline  that  comes  in  after  SE 
or  is  an  integrated  discipline  within  SE?) 


*INCOSE  Handbook,  2007 


NDIA  -  Dr.  Abe  Meilich 
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Observations  on  HSI 


•  “Complexity  is  a  reality  in  systems  design  and  systems 
engineering  and  even  more  so  in  considering  the  multi-faceted 
human  component  considerations  of  a  system”  * 

•  The  Challenge  in  the  design  of  SoS: 

-  Engineers  will  need  to  make  all  new  systems  “net-ready”  for  the 
net-centric  environment,  and  capable  of  working  with  a  wide 
array  of  existing  and  evolving  systems 

•  Each  of  which  were  optimized  for  different  sets  of  human 
interactions  with  their  respective  systems,  which  may  be  at 
odds  for  cross-systems  integration  and  use 

•  Of  critical  importance  is  the  ability  to  integrate  and 
synthesize  data  from  multiple  systems  and  sources  into 
useful  information  that  can  be  employed  by  humans  for 
effective  decision  making  -  i.e.,  the  semantic  interoperability 
problem 


*INCOSE  Handbook  2007 


NDIA  -  Dr.  Abe  Meilich 
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Implication  from  DOTMPLF  on 
System  Engineering 


•  NetCentric  SoS  implies  breaking  organizational  barriers 

•  The  ley  human  element  herets  Trust 

-We  must  not  only  re-engineer  our  system  to  leverage 
NetCenricity,  we  must  also  re-engineer  the  enterprise 

•  Will  the  system  from  organization  A  be  there  (availability  and 
QoS)  to  support  the  system  from  organization  B? 

•  Today  -  Mismatch  between  rate  of  applying  technology 
to  the  problem  versus  the  organizational  and  business 
implication  of  the  transformation 

-All  facets  pf  DOTMPLF  must  be  re-evaluated  when  a  new 
capability  must  be  assessed  against  NetCentric  principles 


NDIA  -  Dr.  Abe  Meilich 
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A pplying  Human  Systems  Integration  (HSI) 


•  Advances  is  practice  of  HSI  are  beginning  to  provide  the  capability 
to  quantify  and  measure  human  characteristics.  These  newer 
methods  also  allow  better  decisions  to  be  made  early  in  the  design 
and  development  process  where  changes  are  relatively 

inexpensive  to  make.”1 

-  Need:  Greater  focus  on  HITL  as  a  measurable  component  of  capability 
MOE  and  its  implementation  in  SoS  (systems  +  humans)  MOP 

-  Trend:  Systems  and  products  that  can  be  operated  and  repaired  by 
fewer  people,  by  lesser  skilled  people,  and/or  people  with  lesser 
training  will  be  in  greater  demand. 

•  Manpower,  personnel,  and  training  (example:  UAV  and 
takeoff/landing  expertise)  are  becoming  key  consideration  in  cost 
effectiveness  and  mission  effectiveness 

•  Need  to  make  the  human  component  an  "inherent  part  of  the 
system,"  and  the  drive  toward  "quantification  of  people  variables" 
in  the  overall  system  engineering  of  the  system  or  SoS 

-  In  an  era  where  “technology  will  solve  the  problem”  it  is  a  challenge 
for  technologists  to  integrate  the  contributions  of  the  soft  sciences  in 
the  design  of  military  systems  that  allow  deterministic  design 
solutions  based  on  physics  or  bits/bytes. 


1  “ Handbook  of  Human  Systems  Integration ”,  Harold  Booher,  Wiley  &  Sons ,  2005 


NDIA  -  Dr.  Abe  Meilich 
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A pplying  Human  Systems  Integration  (HSI) 


«  NetCentric  SoS  -  The  human  is  now  taking  an  active  and  leading 
role  in  combining  systems  (and  or  services)  to  provide  new 
capabilities  (at  run  time  versus  at  design  time) 

B  For  the  HITL  today,  this  is  art;  depending  on  the  COI  and 
availab|ity  of  new  evolving  services. 

-  Can  we  bring  science  to  this? 

-  Passing  Power  to  the  Edge  -  implies  new  training  paradigm  with 
new  SoS  assets  to  configure  and  use. 

-  Commanders  are  challenged  to  plan  tasks  in  hours  vs.  days; 
planning  in  minutes  versus  hours 

•  Paradigm  shift:  Less  decision  and  information  flow  up  and  down 
the  chain  of  command 

•  Commander  now  “shepherds”  or  “monitors”  versus 
“commands” 

•  What  is  the  minimum  information  required  to  make  decisions  at 
the  Edge?  -  An  area  of  intensive  research  and  experimentation 

-  How  do  we  capture  and  analyze  the  impact  of  an  operational 
architecture,  and  its  complementary  system  architectures,  when 
we  are  asked  to  accommodate  responsive,  agile,  dynamic  (on-the- 
fly  changes  to)  operational  approaches  in  a  NetCentric 
environment? 


NDIA  -  Dr.  Abe  Meilich 
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Applying  Human  Systems  Integration  (HSI) 

•  “..lit  can  be  expected  that  HSI  activities  will  become 
more  closely  associated  with  constructive,  virtual,  and 
five  simulations”  U 

-  Measurement  of  Human-in-the-loop  parameters  (primarily 
cognitive  and  social  parameters)  in  the  field  has  been 
problematic  for  SEs  to  define,  apply,  and  measure 

^  Not  enough  time  and  motion  studies  in  battle  as  opposed  to 
measuring  business  processes  in  a  factory 

-  Experimentation,  in  lieu  of  engineering,  has  been  pursued 

•  Where  can  the  contributions  in  decision  theory  help  SE 
a  SoS? 

-  “Blink”  »  What  does  this  tell  us? 

•  Do  people  blindly  look  at  and  process  data  given  to  them  by 
machines  that  will  anticipate  their  needs? 

•  What  filters  are  on?  »  varies  from  individual  to  individual 

-  How  can  this  be  reproduced  and  designed  to? 

-  Is  experimentation  our  only  route  to  understanding? 

1  “Handbook  of  Human  Systems  Integration”,  Harold  Booher,  Wiley  &  Sons,  2005 


NDIA  -  Dr.  Abe  Meilich 
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Observation  from  the  Perspective  of  Operations  Analysts * 


•  Key  challenge  of  SE  of  information  systems  is:  How  do  we  support 
decision  making  to  improve  mission  effectiveness 

-  Decision  making  tasks 

•  How  much  information  is  enough  to  make  a  decision? 

•  The  lower  the  tolerance  for  risk,  the  higher  the  demand  for 
information  to  avoid  that  risk 

•  Commanders  process  information  differently,  therefore, 
information  must  be  shaped  for  the  individual  commander 

-  Tasks  do  not  always  require  quantifiable  information 

-  Just  because  something  cannot  be  measured  or  quantified,  doesn't 
mean  it  isn't  important 

•  Qualitative  methods,  such  as  observation,  have  their  uses  as  well 

-  Commanders  must  perform  their  tasks  in  a  timely  manner 

•  Concern  that  they  will  wait  for  more  or  better  information  rather 
than  act  or  make  a  decision 

•  Need  to  balance  the  need  for  quick  decision  making  with  informed 
decision  making 


Derived  from:  MORS  Workshop  Report:  How  Cognitive  and  Behavioral  Factors  Influence  Command  and  Control, 
Military  Operations  Research  Society,  22  April  2005 


NDIA  -  Dr.  Abe  Meilich 
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Observation  from  the  Perspective  of  Operations  Analysts * 


-  On  the  Network  -  Email,  phone,  and  chat  proliferate  workload 
irrespective  of  the  chain  of  command 

•  Increased  capability  may  decrease  effectiveness  (e.g.,  more 
technology,  information  overload) 

-  Concern  -  In  Network-Centric  Warfare,  everything  depends  on  the 
network  -  What  if  it  doesn't  work? 


Derived  from:  MORS  Workshop  Report:  How  Cognitive  and  Behavioral  Factors  Influence  Command  and  Control, 
Military  Operations  Research  Society,  22  April  2005,  p.22 


NDIA  -  Dr.  Abe  Meilich 
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Considerations  for  Systems  Engineering 


•  System  Engineering  tradeoffs 

-  Drivers  in  the  problem  space 

•  Changing  nature  of  military  operations  (sectarian-based  urban  vs  national 
armies 

•  Call  for  certain  cognitive  and  social  behavior 

-  e.g.,  each  system  optimized  for  human  in  system-specific  domain; 

SoS  human  behavior  subject  of  research 

-  Requirements  in  terms  of  the  human  component  of  the  architecture 

•  As  opposed  to  mission,  task,  or  technology 

-  Examples 

•  Agility  and  adaptability  refers  to  the  human,  rather  than  the 
command  and  control  process 

•  Distributed  collaboration  refers  to  the  people  who  collaborate, 
rather  than  the  tools  used  to  collaborate 

-  Technology  can  work  well,  but  still  not  contribute  to  battlefield 
performance 


NDIA  -  Dr.  Abe  Meilich 
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Considerations  for  Systems  Engineering 


•  Cognitive  engineering  (sometimes  called  cognitive  systems  engineering) 
is  a  multidisciplinary  endeavor  concerned  with  the  analysis,  design,  and 
evaluation  of  complex  systems  of  people  and  technology. 

-  It  combines  knowledge  and  experience  from  cognitive  science,  human 
factors,  human-computer  interaction  design,  and  systems  engineering. 

-  focused  on  how  people  actually  interact  with  complex  technical  systems 

-  Human-computer  interaction  became  a  recognized  field  within  computer 
science 

-  That  is,  design  must  be  based  on  the  observation  and  understanding  of 
system  users  “in  the  wild.”  * 

-  The  inherent  systems  approach  of  cognitive  engineering  means  that  the 
human  user  must  be  understood  in  the  context  of  task,  tools,  and  work 
environment. 

-  In  recent  years,  these  approaches  and  methods  have  been  applied  to 
prevalent  issues  of  information  overload  and  sense  making. 


*  Cognitive  Engineering:  Understanding  Human  Interaction  with  Complex  Systems 
John  R.  Gersh,  Jennifer  A.  McKneely,  and  Roger  W.  Remington, 

Johns  Hopkins  APL  Technical  Digest,  Volume  26,  Number  4  (2005) 


NDIA  -  Dr.  Abe  Meilich 
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Considerations  for  Systems  Engineering 


•  On  the  network,  the  individual  or  group  Goal-based 
performance  is  reached  by  facilitating  information 
transmitting  seamlessly  as  knowledge  to  the  decision 
maker. 

-As  such,  the  human  needs  be  actively  involved  in  information 
transformation  by  combining  his/her  experience  with  available 
information  to  generate  useful  knowledge. 

•  Pushing  information  alone  will  not  accomplish  this 

•  A  fundamental  tenet  of  Net  Centricity  is  that  it  must  be 
ubiquitously  available  on  demand  (available  for  pull). 

•  Therefore,  ultimate  systems  performance  depends  on  the 
human  element  processing 


NDIA  -  Dr.  Abe  Meilich 
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Observations  on  Experimentation 


•  “All  too  often  designers  are  left  with  the  choice  of  assessing  human- 
system  performance  in  expensive  full-mission  simulations  or  by 
estimating  human  capabilities  from  handbooks  and  guidelines.  Neither 
approach  has  proven  satisfactory.  Increasingly,  DoD  and  NASA 
sponsors  have  been  supporting  the  development  of  computer 
simulation  to  explore  joint  human-system  performance.  In  such 
simulations  human  behavioral  characteristics  are  represented  in  a 
computer  model  of  the  operator.” 

•  “the  research  loop  from  understanding  basic  human  information 
processing  to  observing  and  analyzing  technology  developed  in  support 
of  that  understanding  is  intertwined  with  engineering  and  design 
primarily  through  the  development  of  prototypes.” 


*  Cognitive  Engineering:  Understanding  Human  Interaction  with  Complex  Systems 
John  R.  Gersh,  Jennifer  A.  McKneely,  and  Roger  W.  Remington, 

Johns  Hopkins  APL  Technical  Digest,  Volume  26,  Number  4  (2005) 


NDIA  -  Dr.  Abe  Meilich 
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Observations  on  Experimentation 


Challenge  of  extracting  meaningful  data  from 
experimentation  with  HITL: 

•  “Decisioii-making  that  is  rule  or  algorithmically  based  can 
be  modeled  directly  [Augmented  Cognition],  but  error  rates 
should  be  estimated  if  humans  are  involved  in  the  relevant 
decision-making 

-  Implications  for  SE:  FMEA  of  technology  based  on  effects  of 
the  HITL  on  Mission  success 

•  Operational  knowledge  of  human  issues  is  still  weak  in 
many  areas  [of  C2].  Systematic  effort  is  required  for 
organizing  a  consistent  program  for  experiments  on  human 
issues.”1  [described  in  2002  reference;  still  exists  today] 


1  NATO  Code  of  Best  Practice  for  C2  Assessment,  CCRP,  2002 


NDIA  -  Dr.  Abe  Meilich 
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Summary  Comments  on  Engineering  in  the  Cognitive  and^ 
Social  Domains 


•  Observation  and  feedback 

-  HSI  Cognitive  and  Social  -  expensive  if  done  late  in 
development  of  a  system;  experimentation  and  modeling 
needed  at  the  SoS-level  during  CONOPS  development 

•  Stimulus/response  analysis 

-  Paper  simulation 
-War  gaming 

•  Concept  exploration  with  instrumentation 

•  Isolation  of  components  of  cognitive  domain  and  social 
domains 

-  Use  NetCentric  Operations  Conceptual  Framework  as  a 
starting  point 


NDIA  -  Dr.  Abe  Meilich 


22 

10/24/07 


Summary 


•  We  need  to  provide  systems  and  services  to  our  warfighters 
quickly,  but  not  without  understanding  how  they  are  used  both 
cognitively  and  socially  on  the  battlefield 

•  This  is  broader  than  the  Human  Computer  Interface  (HCI)  used  in 
systems  engineering  today 

•  Just  as  bio-engineering  revolutionized  medicine,  Cognitive 
Engineering  and  HSI  as  key  considerations  in  systems 
engineering  are,  and  will,  revolutionize  the  conduct  of  war  on  both 
the  strategic  and  tactical  levels  of  warfare 

•  Learn  more?  Good  references  can  be  found  in  the  INCOSE  SE 
Handbook  in  the  chapter  on  HSI 


We  need  to  move  from  system-centered  design 
to  human-centered  design 
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Agenda 


i  Background 
i  Contents  of  the  Plan 
i  Conclusion 
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Background 


i  More  structured  planning  of  M&S  needed 
i  Evidence  -  Dept.  Air  Force  AFI  16-1002 

-  Modeling  &  Simulation  Support  to  Acquisition 

-  SAF/AQI  “M&S  support  Plan  Template” 
3/29/00 

Some  existing  program  plans  call  for  M&S 

-  placeholder  in  the  document 

Nothing  specific  or  focused  strictly  on  M&S 


Key  Difference:  M&S  as  a  product  versus  as  a  tool 
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Background  (cont) 


Training  community  leading  the  way  - 
mainly  simulation  “products” 

i  Test  community  is  growing  in  its  use  - 
also  mainly  simulation  “products” 

Modeling  on  the  rise  -  typically 
engineering  tools 


System  Level  M&S  Context  hierarchy  gives  rise  to  models  becoming 
end  products  versus  tools  -  see  next  chart. 
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System  Level  M&S  Context  (Hierarchy) 


M&S  Uses 

Concept  Exploration 
Requirements  Analysis 
Performance  Analysis 
Engineering  Design  Trades 
Engineering  Design 
Test  &  Evaluation 
System  Acceptance  A 


WARFARE 
MISSION 
AREA 


FORCE  ON  FORCE 
LEVEL 


PLATFORMISYSTEM 

LEVEL 


SUBSYSTEM/ENGINEERING  LEVEL 


Primary  Target 
Audience 

- Customer 


ystem  Architect 
System  Analyst 

Engineering  Design 


Modeling  and  Simulation  Types 


Use  the  Right  M&S  Tool  For  The  Job 

lY  yf  I'fjSr'  ji  'jrv  j  **  fr*  v *vv  M"r  *'t 6*'  '*/ 


Figure  1 
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Background  (cont) 

i  A  formalized  approach  /  structure  is 
order 

i  Certain  items  need  to  be  addressed 

Will  allow  for  proper  evaluation, 
estimation,  and  tracking 
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Contents  if  the  Plan 

i  Five  sections  (to  discuss) 

-  Executive  Summary 

-  Strategy  /  Approach 

-  M&S  Management 

-  M&S  Activities 

-  Infrastructure 


There  is  a  sixth  section  -  Funding;  breakdown  of  funding  for  effort 
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Executive  Summary 


i  Describe  plan  contents 
i  Describe  M&S  purpose 
i  High  level  description  of  modeled  system 


Range  of  context: 

From  concept,  exploration,  requirements,  analysis, 
training,  and  test  events,  to  a  formal  acceptance 
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Strategy  /  Approach 


i  Describe  M&S  role  in  supporting  Life 
Cycle  process  of  the  program 

i  Identify  method  /  approach  to  be  used  and 
any  special  needs 

i  Describe  the  process  to  field  the  products 
to  be  used 


Reference  any  know  processes  or  procedures  already  in  place 
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M&S  Management 


i  Program  responsibility 

-  Program  level  description  -  who  will  lead 
overall  M&S  effort  down  to  individual 
components 

-  Description  of  Boards  and  Committees 

-  Identify  expected  skill  sets  /  skill  mixes 


All  roles  and  responsibilities  should  be  described 
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M&S  Management  (cont) 

l  CM/DM 

-  Describe  CM  process  -  throughout  the  life 
cycle  (program  most  likely  will  have  CM  plan) 

-  Describe  DM  process  -  throughout  the  life 
cycle 

-  Should  include 

■  Process  to  verify  consistency  /  correctness  of  data 
■Appropriate  representation 

■  Process  to  certify  data 


Reference  any  know  processes  or  procedures  already  in  place 
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M&S  Management  (cont) 

iV&V 

-  Specific  to  M&S  components 

-Accreditation  -  V&V  must  be  explicitly 
identified 

(Direct  connection  to  Accreditation  Plan) 

-  Other  program  plans  most  likely  to  be  affected 


Identify  process  for  comparing  M&S  against  real  data  &  criteria  for  acceptance 
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M&S  Activities 


Development  -  software  development  plan  may  exist 

Data  identification  -  data  flows,  constructs,  metadata 

Releases  -  process  of  M&S  release;  include  security 
and  legal  aspects  (if  applicable);  include  approval 
process  &  procedures,  authority  for  a  release 

i  Events  -  run  for  record,  program  milestones;  include 
external  activities 

i  Tools  -  what  program  teams  need  (analysis  and/or  event 
perspective);  day-to-day  operation,  test 


Describe  and  identify  as  much  as  possible  of  the  above  activities 
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Infrastructure  -  2  parts 

Organizational 

-  Support  -  internal  and  external 

-  Expertise,  training,  connectivity 

i  Facilities 

-  Space,  floor  plans/layouts 

-  Power/HVAC 

-  Computers,  test  equipment,  spare  parts 

-  Security  -  physical  and/or  logical 


Should  include  technology  development  (road  maps)  and/or  strategies 
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Conclusion 


i  M&S  can  end  up  being  a  mini-program  of 
its  own 

i  Processes  same  as  if  it  is  a  program 

This  approach  to  capture  these  items 
provides  focus  on  M&S 


The  hinge  to  continued  future  success  is  in  the  appropriate  use  of  M&S 
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ASC  Engineering  Directorate 


Integrity  -  Service  -  Excellence 


Sound  Systems  Engineering, 
Assures  Proper/Early 


Producibility 

Dr.  Thomas  F.  Christian, 
Richard  Stepler,  Hamid  Akhbari 
Aeronautical  Systems  Center 
NDIA,  10th  Annual  Systems 
Engineering  Conf.  23  Oct.  2007 


Manufacturing  &  Quality  in  Systems  Engineering 


•  Why  are  we  here? 

•  Are  there  really  deficiencies  in  our  Systems  Engineering  Process  or  is  there  a 
problem  in  execution  ? 

•  What  metrics  are  our  critics  using  to  gauge  our  performance? 

•  Failures  to  make  rate,  cost,  and  schedule  in  production 

•  Unresolved  engineering  issues  from  the  development  phase  manifest 
themselves  as  cost  and  rate  failures  detected  during  LRIP  and  Production 

•  Fixing  the  root  cause  of  manufacturing  problems  means  including 
manufacturing  as  a  mandatory  discipline  in  the  System  Engineer  tool  set 
during  design  and  development 

•  My  perspective  on  the  Past,  Present,  and  Future  of  Manufacturing 
and  Quality  involvement  in  Systems  Engineering 


Overview 


•  Where  we  were  ► 


•  Where  we  are  ► 


Where  we  are  going  ► 


=  MLtscum  Photo  Archive 


Where  We  Were:  M&Q  in  America 


Ahhh. . .  .’’The  Good  Old  Days”? 

. Were  they  as  “good”  as  we  remember. . . 

Separation  of  design  and  manufacturing  functions 

•  Transition  to  production  always  problematic 

•  LRIP  created  to  address  problem  but  only  addressed  symptoms 

•  Major  redesigns  of  components  required  to  achieve  desired  production 
rates 

•  Producibility  changes  euphemism  for  “we  can’t  afford  to  build  what 
you  designed” 

•  Cost  high:  low  first  pass  yields  and  traveled  work 

•  Schedule  fluctuations  due  to  excessive  “work  in  process” 

Quality  by  inspection 

•  The  most  expensive  and  least  effective  approach 

•  Build-test- fix-retest- fix-retest - who  pays  for  quality 


Where  We  Were:  M&Q  in  America 


•  Need  proof? 

•  Back  in  the  1970s,  how  long  did  your  domestic  car  last? 

•  Corporate  commitment  to  quality  and  the  customer’s  satisfaction? 

•  “What’s  good  for  General  Motors  is  good  for  the  country” 

•  Then  came  competition  from  Japan,  with  help  from  Deming 

•  The  American  auto  industry  wakes  up 

•  Recognized  Japan’s  focus  on  customer  satisfaction  and  quality 

•  Today:  we  expect  our  cars  to  work  every  time. . . .all  the  time 

•  Toyota  is  still  at  the  forefront  of  quality 

•  Their  “secret”  -focus  on  quality  and producibility  during  design 

THE  W.  EDWARDS  DEMING  INSTITUTE# 
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Where  We  Are:  M&Q  in  America 


FACT:  The  American  Defense  Aerospace  Industry  has  produced, 
unquestionably,  the  finest  weapon  systems  in  the  world 

•  But  at  what  cost?  How  long  does  it  take?  How  much  “cost  of 
quality”  have  we  shifted  to  operations  and  maintenance? 


“...DoD  is  simply  not  positioned  to  deliver  high-quality  products  in  a  timely  and 
cost-efficient  fashion.” 

GAO  (HASC)  Testimony  “Actions  Needed  to  Get  Better  Results  on  Weapons  Systems 

Investments”  5  April  2006 


“In  2001,  The  Average  Weapon  System  Acquisition  Program  Experienced  a 
36%  Cost  Overrun  and  Schedule  Delay  of  Two  Years”  -  Dr.  Marvin  Sambur 


M&Q  in  the  DoD: 

Who  We  Were  Then . and  Who  We  Are  Now 


•  THEN....  1975  to  1985 

•  M&Q  Organization 

•  Represented  at  every  acquisition  level  in  DoD 

•  Had  independent  M&Q  evaluation  of  program  with  veto  power 

•  Large  numbers  of  experienced  people 

•  Career  field  with  opportunities 

•  Effective  Mentoring 

•  Tribal  knowledge  and  well  documented  Specs  and  Standards 

•  NOW . 2001  to  present 

•  Not  represented  at  every  acquisition  level  in  DoD 

•  No  seat  and  no  vote  on  program  readiness 

•  Under-represented  at  most  locations 

•  No  representation  at  some  locations 

•  M&Q  Specs  and  Standards  cancelled 

•  Limited  mentoring  opportunities  and  tribal  elders  retired 


GAO  Findings: 
Production  Maturity 


■  Program  is  not  taking  | - ,  Taking  step$  meet 

steps  to  meet  GAO  production  maturity  criteria 

production  maturity  criteria 


Program  demonstrates 
sufficient  production  maturity 


Program 

JPATS 

ABL 

F/A-22 

JSF 

Global  Hawk 


2003 


n/a 


n/a 


n/a 


JHMCS 
Predator  B 
B-2  RMP 
V-22 


n/a 


n/a 


n/a 


2004 

2005 

n/a 

n/a 

n/a 


2006 


n/a 


n/a 


n/a 


n/a 


Color  ratings  based  on  GAO  opinions 
Source:  GAO  Quick  Look  reports  for 


n/a  =  not  assessed 


Defense  Science  Board  ManTech  Study 


DSB  was  tasked  by  SAF/AQ  to  evaluate  the  ManTech  program 

Released  report  in  February  06 

Much  of  the  report  pertains  specifically  to  ManTech 

Portion  of  the  report  addresses  global  acquisition 
manufacturing  issues 

•  Assessing  program  readiness  for  production . (suggested 

using  the  new  Manufacturing  Readiness  Levels  (MRLs), 
more  on  the  MRLs  Later) 

•  Workforce  Expertise  -  clearly  addresses  the  entire  DoD 
acquisition  workforce 

Complete  DSB  report  is  located  at: 


Defense  Science  Board  ManTech  Study 


Findings: 

•  Manufacturing  talent  in  the  DoD  workforce,  and  its  supporting  industrial 
base,  has  and  continues  to  decline 

•Not  enough  people  (both  at  working  level  and  in  leadership  positions) 
understand  the  processes  involved  in  developing  and  manufacturing  defense 
systems 

Recommendations  to  correct  knowledge  deficiency: 

•  Create  policy  requiring  support  for  programs  such  as  ManTech 

•  Implementation  of  MRLs  as  part  of  DoDI  5000.2 


War-fighter  Expectations 


1 .  Avoid  cost  overruns,  performance  shortfalls,  and 
schedule  slips  typically  manifested  during  production 

2.  Improve  quality  and  avoid  surprises 

3.  Ensure  affordability  and  producibility 

4.  Identify  all  potential  MFG  risks  during  transition  from 
development  to  production  and  establish  risks 
mitigation  plans 

5.  Provide  rapid  response  to  emerging  needs,  e.g., 
readiness  (includes  combat  ops,  surge,  parts  and 
spares,  etc.) 


Swinging  Pendulum  of  Acquisition  Reform 


•Where  we  were  was  not  cheap. .  ..but  it  was  defined 

•Where  we  are  is  neither  cheap  nor  defined 

•The  faster. .better.. cheaper  “acquisition  reform” 
pendulum. . .  for  us  a  wrecking  ball. .  ..left  M&Q 
vulnerable 

•We  can  anticipate  the  return  swing  but  must  not  let  it 
drive  us  back  to  the  old  approach. 


•  Let’s  tailor  the  “return  trip” 


ASC/EN  Response  to  War-Fighter  Concerns 


•  ASC/ENSM  plans  to  conduct  a  360°  evaluation  to  identify  solutions 
and  best  practices 


Requirements  Customers 

(SAF/AQ)  (Users’  Experience) 


Industry  Benchmarking 
(Airlines,  Toyota,  etc.) 


Competitive  Assessment 
(Navy,  Army  Acquisition) 


Prime  Contractors 


Prime  Contractor  Messages 


1 .  Government  acquisition  strategies  do  not  require  an 
in-depth  risk  analysis  for  manufacturing  during  product 
design. 

2.  Government  does  not  specify  the  right  deliverables  in 
their  contracts. 

3.  Benchmark  other  industries  to  get  a  better  picture  on 
MFG  related  issues  during  product  development  and 
risk  mitigation  plans  to  address  them. 

4.  Assess  production  readiness  in  a  meaningful  way. 

5.  More  emphasis  on  suppliers  during  product 
development. 


Prime  Contractor  Messages 


1 .  Government  acquisition  strategies  do  not  require  an 
in-depth  risk  analysis  for  manufacturing  during  product 
design: 

•  Establish  effective  source  selection  criteria  to  emphasize 
producibility  and  affordability 

•  Identify  incentives  for  contractors  to  focus  on  producibility  and 
affordability  during  product  development. 

•  More  MFG/QA  emphasis  during  ASP  reviews 

•  Strong  Government  advocate/champion  are  needed 

•  There  is  a  gross  lack  of  knowledge  and  personnel  in  this  area 

•  Hold  PMs  and  chief  engineers  accountable 

•  Educate  Government  PMs  with  potential  MFG/QA  risks  and 
their  impacts  to  the  overall  system  life  cycle  cost 

•  Make  long-term  decisions  thoroughly  considering  all 
production  risks  down  the  road 


Prime  Contractor  Messages 


2.  Government  does  not  specify  the  right  deliverables  in 
their  contracts: 

•  Government  needs  to  verify  that  the  contractor  has  the  right 
processes  in  place  to  deliver  the  right  product 

•  Government  does  not  use  the  right  metrics  to  measure 
performance 

•  Make  the  contractor  demonstrate  that  they  have  a  solid 
production  plan 

•  Require  the  prime  to  demonstrate  control  of  MFG  processes 
during  development 

•  Specify  proper  MFG/QA  contractual  requirements  in 
development  contracts 


Prime  Contractor  Messages 


3.  Benchmark  other  industries  to  get  a  better  picture  on 
MFG  related  issues  during  product  development  and 
risk  mitigation  plans  to  address  them: 

•  Consider  world-class  performers  in  other  industries 

•  Think  outside  the  box 

•  Develop  lessons  learned 

•  Evaluate  commercial  programs  and  practices  as  well  as  the 
FAA 

•  Consider  having  budget  for  “Producibility  Improvement  Plan” 
(PIP) 


Prime  Contractor  Messages 


4.  Assess  production  readiness  in  a  meaningful  way: 

•  Government  needs  to  develop  better  MFG  transition 
strategies 

•  Willoughby  templates  (Transition  from  Development  to 
Production)  are  useful  tools 

•  PRRs  are  not  focusing  on  the  right  parameters.  Many 
programs  do  not  conduct  full  blown  PRRs  like  they  once  did 

•  MRLs  will  be  a  useful  tool  once  up  and  running 

•  Government  PMs  should  be  required  to  develop  MFG  exit 
criteria  for  milestone  reviews 

•  Industry  recognizes  “Production  Plans”  once  required  by  the 
Government  for  most  programs  as  a  useful  tool 


Prime  Contractor  Messages 


More  emphasis  on  suppliers  during  product 
development: 

The  vast  majority  of  quality  related  issues  come  from  lower- 
tier  suppliers.  Ensure  that  the  prime’s  processes  for 
management  of  their  suppliers  are  solid 

Properly  manage  requirements  flow-down  to  lower-tier 
suppliers 

Require  suppliers  participation  on  IPTs  during  product 
development 

Ensure  supplier  participation  in  the  systems  engineering 
process,  in  particular  MFG  processes  and  procedures 

Develop  predictive  indicators  to  assess  supplier’s  “internal 
health” 

Use  of  common  metrics 


Where  We  Should  Be  Going 


•  The  way  forward . 

•  Internally:  Manufacturing  and  Quality  must  be  the  responsibility  of  design 
engineers  and  be  considered  early  in  the  development  process 

•  Externally:  Supplier  Management. . .  .engineers  at  primes  must  partner  with 
suppliers  to  achieve  maximum  affordability 

•  To  help  with  all  of  it:  The  M&Q  tool  set: 

•  Manufacturing  Development  Guide-Available  now 

•  Manufacturing  Readiness  Levels-Draft  available  now 

•  Manufacturing  and  Quality  Integrity  Program- Available  soon 


Systems  Engineering  -  Ensure  that  design  meets 

requirements  and  is  [producible 

Mfg/QA  helps  SE  meet  producibility,  OSS&E,  and  Airworthiness 

Design  requirements 


M&Q  Tool  Set 

Manufacturing  Development  Guide 


Best  Practices 


http :  / /  engineering .  wpafb .  af .  mil/ mdg/mdg .  asp 


Mfg  Capability/Risk  Mgt 


Production  Cost  Modeling 

Key  Suppliers 

Key  Characteristics  and 
Processes 

Variability  Reduction 

Virtual  Mfg 

Design  Trade  Studies 

Product/Process  Validation 

Process  Control  and  Cont 
Improvement 


•  Developed  by  a  joint  industiw 
Government  team,  improved  over  the  past 
10  years 


•  Recognized  aerospace  industry  guide  for 
describing  the  role  of  Manufacturing  and 
Quality  in  the  Systems  Engineering 
process 


•  Available  at: 

http  ://engineering.  wpafb .  af.mil/mdg/mdg. 
asp 


Factory  Efficiency 


DMSMS 


M&Q  Tool  Set 

Manufacturing  Readiness  Levels 


Defense  Acquisition  Life  Cycle  Framework 


Concept 

Technology 

System  Development 

Production  and 

Operations  and 

Refinement 

Development 

and  Demonstration 

Deployment 

Support 

Design 

FRP 

Concept 

Readiness 

Decision 

Decision 

Review 

Review 

Technology  Readiness  Levels 


TRL  1-3 

TRL  4 

TRL  5 

TRL  6 

TRL  7 

TRL  8 

TRL  9 

Manufacturing  Readiness  Levels 

MRL  1-3 

MRL  4 

MRL  5 

MRL  6 

MRL  7 

MRL  8 

MRL  9 

MRL  10 

MRLs  address  topics  such  as: 


•  Parallels  Technical  Readiness  Levels 

•  OSD  planning  to  make  MRL  assessments  a  milestone 
requirement 

•  Available  at: 

https://acc.dau.mil/CommunityBrowser.aspx?id=l  823 1 

Producibility 

Key  Characteristics 
Material  Availability 
Production  Simulation 
Process  Controls 
Tooling  p/ 

M&Q  Tool  Set 
M&Q  Integrity  Program 


This  DRAFT,  dated  25  August  2006,  prepared  by  AF-1 1, 
has  not  been  approved  and  is  subject  to  modification. 

DO  NOT  USE  PRIOR  TO  APPROVAL. 

(Project  SESS-2006-001) 

DEPARTMENT  OF  DEFENSE 
HANDBOOK 

MANUFACTURING  AND  QUALITY 
INTEGRITY  PROGRAM 


AMSC  XXXX  AREA  SESS 


NOT  MEASUREMENT 
SENSITIVE 


MIL-HDBK-DRAFT 

DATE 


•  Contains  the  practices  described 
in  the  Manufacturing 
Development  Guide 

•  Includes  suggested  contractual 
and/or  Systems  Engineering  Plan 
requirements  and  verifications 

•  Will  be  posted  on  the  ASC/EN 
public  Website 


Producibility. .  .M&Q’s 
Contrib ution  to  Systems  Engin eer ing 

America’s  Defense  Aerospace  Industry  is  #1  in  the  world. 

However. . . 

•  Are  we  be  able  to  buy  desired  quantities? 

•  How  many  B-2s  were  originally  planned?  F-22s?  JSFs? 

If  we  can’t  figure  out  a  way  to  build  better  systems  cheaper,  we  will 

fulfill  Norm  Augustine’s  prophecy: 

•  “In  the  year  2054,  the  entire  defense  budget  will  purchase  just  one  aircraft.  This 
aircraft  will  have  to  be  shared  by  the  Air  Force  and  Navy  3-1/2  days  each  per 
week  except  for  leap  year,  when  it  will  be  made  available  to  the  Marines  for  the 
extra  day.  ” 

My  primary  focus  is  to  integrate  quality  and  producibility  early  in  the 

Systems  Engineering  process  (see  the  ASC,  MDG) 


Encouraging  Steps  Forward 


Tri  services  committee  being  formed  at  the  SES  level 


NDIA  committee  on  “  Manufacturing  &  Quality  Assurance  formed 
as  govemment/industry  forum. 

•  If  you  share  my  interest  in  this  subject,  join  me  on  the  committee 


Fixing  the  root  cause  of  manufacturing  problems  means 
including  manufacturing  as  a  mandatory  discipline  in  the 
Systems  Engineering  tool  set  during  design  and 
development 


Summary:  We  Know  We  Are  There  When 


•  The  process  used  to  manufacture  a  part  is  given  equal 
consideration  as  the  functionality  of  the  part 

•  Product  performance  and  producibility  are  equal  in  the  risk 
analysis  trade  studies  during  product  development 

•  The  “chiefs”  of  design,  manufacturing  and  logistics  have  equal 
votes  on  critical  design  decisions  during  development 

•  “Design”  executives  are  held  accountable  for  unit  production  cost 
and  cost  of  quality  decisions 

•  M&Q  metrics  are  present  in  entry  and  exit  criteria  for  each  phase 
of  the  acquisition  life  cycle 

•  Integrating  people,  processes,  and  technology  using  the  System 
Engineering  Process  proven  effective  by  world  class  producers 


As  good  systems  engineers,  our  commitment  to 

M&Q  starts  in  design 


Thank  you  for  your  time 
and  attention 

Thomas.Christian@wpafb.af.mil 

Phone;  937-255-1826 


Implementing  a  Systems 
Engineering  Risk 
Program  in  a 
Sustainment 
Environment 


Jim  Miller 

Chief  Engineer 

727  ACSG/EN 

Phone:  (405)  736-7996 

james.c.miller@tinker.af.mil 


What  Sustainment  Environment? 


727th  Aircraft 
Sustainment 
Group 

Col.  James  Fulton 
Commander 

Ms.  Jerri  Hulme 

Deputy  Director 

Mr.  James  Miller 

Chief  Engineer 


PROVIDING  EFFECTIVE  &  EFFICIENT  WEAPON  SYSTEM  SUPPORT 


727  ACSG  Mission 


Single  Manager  for  Sustainment  and  Modernization  of 
400+  USAF  Commercial-Derivative  Aircraft 
HF  Global  Communications  System  Network 

Preserves  FAA  Certification  and  Operational  Safety, 
Suitability  &  Effectiveness  (OSS&E)  of  Commercial 
Derivative  Aircraft 

4  Squadrons  Manage  Services  Acquisition 


Weapon  System  Support 


727th  Aircraft  Sustainment  Group 


Contractor  Logistics  Support  (CLS) 


Customers 


Weapon  Systems 


AMC 

ACC 

ANG 

AFRC 

AETC 

USAFE 

PACAF 

AFMC 

USAF  ACADEMY 

AF  FLIGHT  STD 

AGENCY 

ARMY 

NAVY 

US  MARINE 

CORP 

DIA 

DSCA 

FMS 

USSOCOM 


KC/KC 

VC-25 

E-4B 

C-9 

C-12 

C-20 

C-21 

C-26 

C-38 

E-9 

T-41 

T-43 

T-51 

TG-10 

TG-15 

UV-18 

Peace 

HFGC 
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727  ACSG  Responsibilities 


r400+  Active 
41  Inactive 
Aircraft  Mgd 


19 

Commands 


Weapon 

Systems 


56  USAF 
Bases 
2  FMS 
Nations^ 

Ain raw 


FY07 
50  PDM 
Scheduled 


FY07 
$913M 
Obligation 
L  Authority 


FY07 

$6.6B 

Contracts 


Weapon  System’s  Missions 


Presidential 
Airlift 
“Flying 
White  House 


Highly 
Survivable 
NMCS  Node 
lying  Pentago 


Tanker  ^ 
Aerial  Refueling  ^ 
&  Airlift 
Support 


Intelligence 

Surveillance 


Diplomatic, 
VIP&  DV 
Travel 


Recon 

(ISR) 


Special 

Duty 

Support 


HF 

Communications 


Counter-drug 

Support 


Seasoning 


|  Sea  Surveillance  1 

Pilot  &  Navigator 

B\iv 

{  Radar  &  HF  fl 

K  Training  & 
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MEDEVAC 

So  What  is  the  Problem? 


•  Numerous  classes,  training,  &  regulations  on  risk 

-  Most  aimed  at  acquisition,  not  sustainment 

-  No  detailed  direction  for  a  workable,  grass-roots 
approach 

•  Sustainment  different 

-  Not  one  big  pass/fail 

-  Most  new  risk  associated  with  mods  or  unique  events 

•  Our  organization  had  insufficient  direction, 
documentation,  and  procedures  to  implement  an 
effective,  comprehensive  Systems  Engineering 
risk  program 


So  What  Are  Doing  About  It? 


•  Instigated  a  step-by-step  Operating  Instruction 
to  implement  risk  management  throughout  the 
organization 

•  Trained  the  workforce  for  common  SE  baseline 

•  Implemented  tangible  approach  that  is: 

-  Aimed  at  the  working  level 

-  Applicable  throughout  entire  organization 

-  Accounts  for  progress  through  metrics 

-  Always  starts  with  requirements 


Workforce  Training 


Courses  Selected  for  First  Year  (All  CBT): 

•  SYS  182  Intro  to  Systems  Engineering  ~  3  hrs 

•  SYS  172  Modification  Management  Process  ~  6  hrs 

•  SYS  155  Operational  Safety,  Suitability  &  Effectiveness  ~  9  hrs 

•  SYS  028  Intro  to  Configuration  Management  ~  16  hrs 

•  SYS  165  Intro  to  Risk  Management  ~  21  hrs 


Who:  All  PM’s,  Equipment  Specialists  and  Engineers 
When:  Complete  in  12  months 


Workforce  Training  Metric 


Org  A  Training  Progress  (45  People) 
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Risk  Management  Process 


I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

I — |  RMT 


Figure  1.  Flowchart  for  Risk  Management  Process 


Step  1:  Build  a  Risk  Management  Team 


•  Program  Manager  formally  establishes 
RMT  in  writing 

•  RMT  consists  of,  at  a  minimum: 

-  Program  Manager 

-  Project  Engineer 

-  Representative  from  the  customer 

-  Representative  from  the  contractor 


I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

□  RMT 


Figure  1 .  Flowchart  for  Risk  Management  Process 


Step  2:  Review  Requirements  Correlation  Matrix 


•  Review  established  RCM 

-  Review  all  initial  identified  risks  assessments 

-  Verify  initial  assessment 

-  Determine  if  all  risks  have  been  identified 


I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

□  RMT 


Figure  1 .  Flowchart  for  Risk  Management  Process 


Define  Requirements 


•  Break  requirements  down  in  a 
Requirements  Correlation  Matrix  (RCM): 

•  Spreadsheet  with  following  columns: 

-  Requirement 

-  Requirement  Source 

-  Derived  Requirements 

-  Quantification 

-  Operational  Conditions 

-  Initial  Risk  Assessment 

•  Give  RCM  to 

-  Test  Team  for  their  planning 

-  Risk  Mngt  Team  for  their  planning 


Req 

Req 

Derived 

Req 

Op 

Risk 

Title 

Source 

Req 

Definition 

Quantification 

Cases 

(R/Y/G) 

Step  2:  Review  Requirements  Correlation  Matrix 


•  Review  established  RCM 

-  Review  all  initial  identified  risks  assessments 

-  Verify  initial  assessment 

-  Determine  if  all  risks  have  been  identified 


I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

□  RMT 


Figure  1 .  Flowchart  for  Risk  Management  Process 


Step  3:  Periodic  RMT  Meetings 


•  Project  Engineer  shall  chair  meeting 

-  Determine  frequency 

-  Not  less  than  quarterly  to  support  Weapon 
System  Review  (WSR) 

•  Purpose  to: 

-  Review  all  risks 

-  Review  all  mitigation  plans 

-  Identify  new  risks 

-  Build  and  update  quad  charts 

-  Close  risks 


Step  4:  Identify  New  Risks 


•  Identify  any  new  technical  risks 

-  Documented  requirements 

-  Other  sources 


•  RMT  can  provide  input  back  to  RCM 

•  Other  opportunities  to  identify  risks 

-  Event  driven  technical  reviews 


-  Program  management  reviews 


-  Design  reviews 


I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

□  RMT 

Figure  1 .  Flowchart  for  Risk  Management  Process 


Step  5:  Analyze  Old  &  New  Risks 


•  Analyze  each  identified  risk 

•  Use  5  by  5  matrix 

-  Consequence  if  occurs 

-  Likelihood  of  occurring 

•  Perform  Root  Cause  Analysis  for  all  “red” 
and  “yellow”  risks 


Step  6:  Build  Metrics 


•  Two  minimum  metrics 

-  Risk  Assessment  Matrix 

-  Risk  Mitigation  Plan  Roll-up 

•  Quad  Chart  for  each  yellow/red  risk 

•  Metrics  shown  to  management  at 
quarterly  Weapon  Systems  Review 


I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

□  RMT 

Figure  1 .  Flowchart  for  Risk  Management  Process 


Probability 


Risk  #1  Assessment  Matrix 


High 


Technical  Risk:  If  software  complexity 
increases  on  MCS  then  failure  of 
modifications  could  result. 


Impact 

Negligible  *  Critical 


100% 


I 


Risk  Workshop  Completed 
14  Mar  07 


Mitigation  Plan: 

•  Contractor  is  currently  Capabilities 
Maturity  Model  Integration  (CMMI) 
software  level  3  certified  and  has  plan 
to  reach  level  5  by  contract  award 

•  Government  will  ensure  contractor 
will  work  with  ground  agencies  to 
ensure  software  is  interoperable 

•  Government  will  follow  disciplined 
requirement  matrix  process  outlined 
in  727  ACSG  Operating  Instruction 
(O.l.)  to  prevent  unplanned 
requirements/complexity  increases  & 
track  via  established  metrics 


Rick  Quad  Chart 


Risk  Title 

Risk  Tracking  Number 


Background 

Description  of  prob*. 

•  Item  1 

•  Item  2 

•  Item  3 


Risk 

Color 

Code 


J 


Risk  Mitigation  Plan 

•  Proposed  solution  for  implementation 
and  risk  mitigation. 


0 

X 

Actions  to  Date 


Future  Action 


Date 


Established  Risk  Assessment  Date  1 
Completed  Mitigation  Plan  Date  2 

Completed  details  of  mitigation  Date  3 
incorporation  with  contractor 
Received  effort  impact  (cost  and  Date  4 
schedule) 


Proj.Date 

Contract  Award  for  implementation 

Date  1 

Mitigation  Plan  Completion  (or  Date  2 
any  significant  milestones) 

Etc... 


Risk  Mitigation  Plan  Roll-Up 


Step  7:  Develop  Risk  Mitigation  Plan 


•  Risk  Mitigation  Plan  shall  consider: 

-  Cost 

-  Schedule 

-  Safety 

-  Effectiveness 

•  Plan  should  delineate: 

-  Definite  courses  of  action 

-  Address  the  root  cause 

•  Plan  should  be  timely: 

-  Within  14  days  for  high,  or  red,  risk 

-  Within  60  days  for  medium,  or  yellow,  risk 


Step  8:  Implement  Risk  Mitigation  Plan 


•  Program  Manager  will: 

-  Work  with  contractor  and/or  customer  as 
applicable 

-  Incorporate  into  Integrated  Master  Schedule 

-  Budget  funds  accordingly 

-  Schedule  technical  interchange  meetings  as 
required 


Periodic  RMT 
Meetings 
Para  2.1.3 

Identify 
New  Risks 
Para  2.1.4 


RMT  Closes/\ 
Downgrades  Risk 
\  Para  2.1.9  W 


Document 
Lessons  Learned 
Para  2.1.14 

I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

□  RMT 


Figure  1 .  Flowchart  for  Risk  Management  Process 


Step  9:  Update  Mitigation  Plans/Quad  Charts 


•  Project  Engineer  will  update: 

-  Risks 

-  Risk  Mitigation  Plans 

-  Quad  Charts 

-  Metrics 

•  Quad  charts  and  metrics  will  be  briefed  at: 

-  Weapon  System  Reviews 

-  Program  Management  Reviews 

-  Others  as  determined  by  PM 


Step  10:  Unbiased  Review 


•  Project  Engineer’s  boss  will  set  up 

•  Review  will  include  at  a  minimum: 

-  Chief  Engineer 

-  Program  Manager 

-  SMEs  within  the  organization 

-  SMEs  outside  the  organization 

•  Will  go  thru  all  risks,  mitigation  plans, 
quad  charts,  and  metrics 

•  Project  Engineer  will  incorporate  feedback 
from  review 


Step  11:  Track  Metrics,  Charts,  Reports 


•  Project  Engineer  and  Program  Manager 
will  update  throughout  process 

•  PM  will  ensure  information  reported  in 
various  venues  (WSRs,  PMRs,  etc) 


I  I  Program  Manager 
I  I  Flight  Director 
|  |  Chief  Engineer 

□  RMT 


Figure  1 .  Flowchart  for  Risk  Management  Process 


Step  12:  Lessons  Learned 


•  Risks  are  not  snowflakes 

•  Mitigation  Plans  are  not  either 

•  Lessons  Learned  repository  contains: 

-  Possible  risks  to  consider 

-  Potential  mitigation  plans 

•  Repository  is  not  program  specific,  but  for 
entire  organization 

•  Future  plans  are  to  make  the  lessons 
learned  repository  a  database  with 
keyword  searches 


What’s  Next 


Continue  implementation  throughout 
organization 

Continue  Measure/Track  results 
Populate  Lessons  Learned  database 
Refine  as  needed 
Document  successes 


-  We  are  having  some! 


Risk  Management  can  be  implemented,  applied 

AND  make  a  difference 


Summary 


•  727th  ACSG  developed  grass-roots  means  to 
implement  Risk  Management  as  part  or  our 
Systems  Engineering  in  Sustainment 
Environment 


•  Clear-cut,  tangible  processes  steps  for  the 
working-level 


•  Metrics  to  measure  progress  for  management 

•  It  works 


I  n  Place  and  I  n  Use  Now 


Questions  ? 


Questions  ? 


Major  Modification  Programs 


17  Current  Programs 

Q  KC-10  AMP  -  ASC  Lead  (ACAT  II) 
g  KC-10  Dual  406  MHz  ELT  Upgrade  (ACAT  III)* 
g  KC-10  Iridium  Phone  (ACAT  III)* 
g  KC-10  UHF  SATCOM  Antenna  (ACAT  III)* 
g  VC-25  Forward  Lower  Lobe  (FLL)  Cooling  (ACAT  III) 
g  VC-25  Presidential  Data  System  (PDS)  (ACAT  III)* 
g  VC-25  C NS/ATM  (ACAT  III)* 
g  C-20  Gulfstream  Test  Vehicle  (GTV)  (ACAT  III)* 
g  E-9  Telemetry  Sys  Upgrade  (ACAT  III)* 
g  E-4B  Mod  Block  I  (ACAT  II)  * 

G  E-4B  256  Kbps  High  Speed  Data  via  INMARSAT  (ACAT  III)* 
R  C-12EFIS  (ACAT  III) 

D  HFGCS  Network  Control  Station  -  West  (ACAT  III)* 

D  HFGCS  AFSPC  Test  Range  HF  Modernization  (ACAT  III)* 

G  HFGCS  Network  Optimization  -  Spiral  II  (ACAT  III)* 

G  HFGCS  Navy  Consolidation  (ACAT  III)* 

G  HFGCS  Audit  Log  Upgrade  (ACAT  III)* 


$1.03B 

$2.4M 

$2.7M 

$2.6M 

$14.4M 

$223. 3M 

$41. 8M 

$8.7M 

$5.9M 

$421 ,4M 

$8.4M 

$77.7M 

$23. 2M 

$3.9M 

$7.1  M 

$6.4M 

$189K 


Phase  4:  Identify  and  Define  Processes 


Basic  Systems  Engineering  Process 


Requirements 

Analysis 


Verification 

Loop 


Requirements 

Loop 

Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 


Systems  Engineering  Implementation  Phases 


•  Phase  1 

•  Phase  2 

•  Phase  3 

•  Phase  4 

•  Phase  5 

•  Phase  6 

•  Phase  7 


Awareness  of  Need 
Workforce  Training 
Identify  Applicable  Programs/Orgs 
Identify  and  Define  Processes 
Incentivize  Contractors/Partners 
Develop  Library  of  Tools 
Track  Progress  via  Metrics 


Phase  6:  Develop  Library  of  Tools 


□Need  good  SE  “toolbox” 

-  Templates 

-  Metrics 

-  How-to’s  (fishbone,  5-whys,  paredo, ...) 

-  Lessons  Learned 

-  Explanations 

-  Best  Practices 

-  Peer  Review 

-  Case  Studies 

-  Life  Cycle  Cost  consideration 

-  Contractual  language 

-  Etc... 


Functional  Office  to  Develop/Obtain. ...Not  Started  Yet 


Phase  4:  Identify  and  Define  Processes 


Reviewed 


Sample  Organization  Sys  Eng  “Dashboard” 


Sample  Program  Sys  Eng  “Dashboard” 


Requirements  5(K.  Training 


High 

Med. 

Low 


25  V 


Contracts 


Meas. 

Curr. 

Lean 

Doc. 


i'll1 

■  ■ 

i  r  p 

1 1  ill 


Low 


Med.  High 
Risk 


Processes 


Risk  Assessment  Matrix 


Probability 


Technical  Risk  #2 


Low  X  Med 


Impact 


Negligible 


Critical 


Risk  Workshop  Completed 
14  Mar  07 


Technical  Risk:  If  contractor  fails  to 


adequately  perform  systems  engineering 
then  modifications  and  upgrades  could  be 
impacted/delaved. 


Mitigation  Plan: 

•  Contractor  has  Quality  Assurance 
Plan  and  Program  Management  Plan 
on  current  contract.  Plans  will  be 
updated  for  new  contract 

•  Government  will  require  contractor 
to  submit  requirements  correlations 
matrix  (RCM)  for 
modification/upgrade  efforts 

•  Government  will  require  contractor 
to  use  an  approved  risk  management 
program  for  modification/upgrade 
efforts 

Government  will  follow  disciplined 
requirement  matrix  process  outlined 
in  727  ACSG  O.l.  to  prevent 
unplanned  requirements/complexity 
increases  &  track  via  established 
metrics 


Probability 


Technical  Risk  #3 


High 


Impact 

Negligible  *  Critical 


100% 


Technical  Risk:  If  configuration 
management  for  communication 
equipment  is  not  maintained  then  system 
interoperability  could  be  hindered. 


Mitigation  Plan: 

•  Government  will  require  current 
contractor  configuration  management 
plan  will  be  updated  for  new  contract 

•  SPO  will  work  with  users  and 
contractor  to  ensure  regular 
configuration  inventories  are 
occurring  to  ensure  configuration 
reports  are  accurate 

•Government  will  conduct  test 
planning  criteria  and  resource 
requirements  at  the  start  to  minimize 
potential  interoperability  conflicts 
and/or  oversights 


Probability 


Technical  Risk  Summary 


Risk  Workshop  Completed  - 
14  Mar  07 


Probability 


Program  Risk  Summary 


OVERALL  PROGRAM 
RISK  IS  LOW 


Risk  Workshop  Completed  - 
14  Mar  07 


Tracking  Progress  via  Metrics 


S  Metrics  developed  to  track  progress 
S  Metrics  shown  regularly  to  upper  management 

□  1st  staff  meeting  of  month 

Quarterly  Weapon  System  Reviews 
Metrics  must  be  able  to  roll  up 

S  Metrics  will  track: 

Systems  Engineering  Implementation 
Requirements 
v^Risk 
Processes 
S  Training 
Contracts 


Headquarters  U.S.  Air  Force 


Integrity  -  Service  -  Excellence 

Applying  Systems  Engineering 
during  Pre-Acquisition  Activities 


NDIA  10th  Annual 


Engineering 


Division  Conference 
San  Diego,  CA 
23  October  2007 


Jeff  Loren 
MTC  Technologies,  Inc. 

(SAF/AQRE) 


Integrity  -  Service  -  Excellence 
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■  Need  for  Early  SE 

■  Defining  Early  SE 

■  SMC  Pilot  Program 

■  Policy  Initiatives 

■  Challenges 

■  Way  Forward 

Integrity  -  Service 


Outline 


-  Excellence 
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Why? 

“Systems 
Engineering 
is  broken | 
go  fix  iqlg^ 

Attributed  to  SecAF  James  Roche,  spring  2002 


Lack  of  systems  engineering  has  been  cited  as  the 
cause  of  major  defense  acquisition  program  failures 

Cost  overruns,  schedule  slips,  mishaps,  external  criticism, 
instability  in  requirements  and  funding,  poor  acquisition  strategies 


Integrity  -  Service  -  Excellence 
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The  Need 


■  RAND  Project  Air  Force  study:  “Is  Weapon  System 
Cost  Growth  Increasing?” 

"...  despite  the  many  acquisition  reforms  and  other  DoD  management 
initiatives  over  the  years,  the  development  cost  growth  of  military 
systems  has  not  been  reduced.  ”* 

■  This,  however,  does  not  indicate  we  are  necessarily 
doing  badly  ... 

“There  is  no  doubt  that  the  systems  developed  in  each  successive 
decade  are  more  complex  than  those  of  the  prior  decade.  The  ever- 
increasing  complexity  of  technology,  software  density,  system 
integration  complexity,  and  the  like  make  estimating  a  total  system's 
development  cost  ...an  ever-increasingly  challenging  endeavor.”* 

■  Increasing  complexity  has  kept  stride  with 
increasingly  improved  acquisition 

*  Is  Weapon  System  Cost  Growth  Increasing,  2007,  RAND 


Integrity  -  Service  -  Excellence 


Why  It’s  Needed  —  Early  Decisions 
Are  Key  Life  Cycle  Cost  Drivers 


Cumulative  LCC 

Cost  to  Fix 
100% 


10000X 


75% 


1000X 


50% 
100X  " 


10*>5% 


X  - 


Percent  of  Baseline  LCC  Incurred 
Percent  of  Baseline  LCC  Committed 
—  —  Cost  to  Identify  &  Resolve  a  Defect,  and  Incorporate  Change 


888S 


Development 


Integration  Verification  Fielding  Operation 


Concept 

Refinement 


Technology  Sys.  Dev.  p 
\/ Development  and  Demo. 


Production 


Operations  and  Support 
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Defining  Early  SE 


■  What  it  is: 

■  The  systems  engineering  (SE)  tie  between  JCIDS 
and  the  AoA  ...  and  beyond 

■  A  disciplined  process  for  scoping  capability 
needs  and  developing  concepts 

■  The  process  required  to  do  the  necessary  work 
for  a  successful  AoA 

■  A  means  to  identify  candidate  technologies  and 
assess  the  realism  for  transition 

■  An  actual  pre-acquisition  effort 

■  What  it  is  not : 

■  An  AoA 

■  "Gaming  the  system"  to  favor  a  solution 


Integrity  -  Service  -  Excellence 
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Pre-Acquisition  “Systems  Thinking” 

Boundary  Conditions 

Pre-Acquisition  SE  efforts,  like  those  throughout  the  rest 
of  the  life  cycle,  are  essentially  an  “integrating  function” 

■  Pre-A  SE  mainly  occurs  in  two  domains,  each  with 
set  boundaries 

>  The  first  SE  domain  spans  the  period 
from  JCIDS  initiation  of  a  need  to  AoA 
entrance: 

>  The  second  domain  continues  the  SE 
functions  after  the  AoA  until  formal 
program  handoff: 

■  The  SE  functions  in  both  domains  are  fundamentally 
similar,  but  there  are  attributes  unique  to  each 


/AoA  Entrance 

F1(SE)dSE 

DS 

Program  Initiation 

F2(SE)dSE 

AoA  Exit 
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Pre-Acquisition  Example 


Capability  need:  “Get  people  and  equipment 

across  a  body  of  water” 

■  First  pass  asks  key  questions: 

■  What  does  “water”  mean?  (Solution  sets  will  be  very  different 
for  Piscataway  Creek,  the  Potomac  River,  and  the  Pacific  Ocean.) 

■  Are  there  any  obvious  constraints?  (Sensitivity  to  water 
exposure?  Time-in-transit  limitations?) 

■  Initial  analysis  should  yield  various  methods,  and  a  cost  / 
risk  summary  for  each 

■  Airlift  ■  Drive  around  (depends  on 

■  Bridge  total  distance,  thus  time) 

■  Catapult  (unsuitable  for  people)  ■  Ferry 

■  Drive  across  (depends  on  ■  Helicopter 

depth,  current,  etc.)  ■  Tunnel 

■  Analysts  should  also  be  able  to  quickly  rule  out  candidates 
that  don’t  meet  constraints 


Integrity  -  Service  -  Excellence 
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Pre-Acquisition  Example  (cont) 

■  Parametric  trades  within  a  method  (bridge,  tunnel,  etc.) 
consider  how  relevant  factors  (depth,  width,  current,  etc.) 
affect  a  baseline  candidate  solution 

■  “A  mile  upstream  the  channel  is  narrower.  The 
shorter  span  means  ~30%  less  material  cost,  but 
road  access  and  construction  staging  are  difficult. 

■  “A  mile  downstream  the  current  is  slower.  The 
longer  span  means  ~20%  more  material  cost,  but 
you  can  complete  construction  earlier.” 

■  Once  the  AoA  looks  at  families  of  candidates  and  concludes 
that  a  bridge  is  the  best  solution,  a  similar  process  is 
employed  to  determine  the  optimum  type  (cantilever, 
suspension,  pontoon,  single-  or  two-span  draw,  etc.) 

■  Pre-AoA  measures  are  high-level  programmatic  /  operational 
parameters  (cost,  schedule,  vehicle  capacity,  etc.) 

■  Post-AoA  measures  have  a  more  traditional  design  and 
execution  focus  (EVM,  weight,  material  durability,  etc.) 


Reference 

location 
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SE  Applied  to  a  Product  or  System 

Transforming  Requirements  to  Design 
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SE  Applied  to  a  Capability 

“ Requirements  Engineering” 
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SMC  Pilot  Program 


■  Modeled  after  test  case  developed  for  a  relatively 
cheap,  ill-defined  launcher 

■  Study  commissioned  by  SAF/AQR 

■  Objective:  Develop  and  validate  a  concept 
development  systems  engineering  process  &  guide 

■  Identify  barriers  to  success  in  concept  development 

■  Used  standard  systems  engineering  tenets  as  a 
baseline 

■  Modified  for  future  concept  development  efforts 

■  Currently  validating  and  documenting 


Integrity  -  Service  -  Excellence 
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SMC  Concept  Development 

Process  V-Chart 


Identify  & 
Define 
Shortfalls 

Collect 

Ideas 

\: 

Release. 

Approval^ 


M  Final  Conce| 
Review 


Reqs 

Verification/ 

Capabilities 

Assessment 


Acquisition 
Objectives 
&  Costing 


/& 


(f 


System  Characterization 


Key  Sub-System 
Characterization 


Initflj  Concep 
\  ^^view 

* 

System 

\ 

Characterization 

Concept  Characterization 
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NRC  Pre-A  SE  Study 

Co-Chairs:  Dr  Kaminski,  Gen  (ret)  Lyles 

TASKS 

■  Assess  the  contribution  of  pre-A  SE  on  Air  Force  programs 

■  Determine  level  of  pre-A  SE  required  for  program  success 

■  Determine  current  barriers  to  pre-A  implementation,  both  on 
concepts  leading  to  an  AoA  and  for  the  post-AoA  selected 
alternative(s) 

■  Develop  a  framework/methodology  for  developmental 
organizations  to  ensure  proper  pre-A  SE  is  accomplished 

■  Recommend  changes  to  enable  adequate  pre-A  SE,  and  the 
means  for  seamless  transition  from  need  identification  through 
program  office  standup 

STATUS 

■  Study  committee  received  approx  30  formal  briefings 

■  Committee  members  currently  conducting  analyses  and  writing 
assigned  sections  of  report 

■  Anticipate  start  of  peer  review  Jun  07 

■  Public  release  of  final  report  anticipated  Nov  07 
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AF  Early  SE  Policy  Initiatives 


“Nothing  in  the  world  is  more  important 

than  policy!”  * 

■  “Fixing”  SE  in  the  pre-Acquisition  world  requires  a 
two-pronged  approach 

■  Acquisition  policy  -  DoD  5000.2,  63  series  AFIs 

■  Requirements  policy  -  JCIDS,  AF1 10-601 

■  Current  policies  encourage  acquisition  and 
requirements  to  coordinate,  but  do  not  have  hooks 
to  force  working  together 

■  Islands  of  success  exist,  but  tend  to  be  personality/ 
experience  driven 

■  Opportunities  exist  to  slip  early  acquisition  community 
SE  involvement  into  the  requirements  process 

*  Lt  Col  Mark  Wilson,  Policy  Branch  Chief,  19  Oct  07 
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SE  and  Technical  Planning  in 
Pre-Program  Concept  Development 


MAJCOM 
Requirements 
(Operational 

Needs)  Concept 


^A^toncept  4 


MAJCOM-led 

AoA 

Concept  1 
Concept  3 


Product  Center 
‘Proto-SPO’VCadre 


Concept  3 


SAF/AQ  provides  guidance  to  Product 
Centers  for  CDP*,  used  to  decompose 
needs  into  requirement  sets/concepts 

DEFINING  THE  SOLUTION  SPACE 


SAF/AQ  or  Product  Center/CC  reviews 
AoA  Study  Plan  prior  to  MDA  approval, 
to  ensure  Product  Center’s  technical 
planning  for  each  concept  has  been 
accomplished  per  their  CDP*  and 
documented  in  a  Concept  SEP 


SAF/AQ  provides  guidance 
to  SPO  Cadre  for  SEP  used 
during  concept  maturation 


CDP  -  Concept  Development  Process 


A 

▲  t  A  PEO  1 

A  ^  r 

A 

MSI  '  MSI  Program  3 

KDP  A  PSR  KDPB  P' 

„D  MS/  A  o&s  / 

5K  KDP  C y/  Sustainment 

SAF/AQ  or  Product  Center/CC  conducts 
“Concept  Technical  Review” 

(~  equivalent  to  PSR  or  Sufficiency  Review) 


SAE  provides  guidance  on  SEP 
and  program  oversight 


Integrity  -  Service  -  Excellence 


16 


Challenges 


■  Begins  with  a  “R”,  rhymes  with  “forces” 

■  Experience  “bathtub”  (lots  of  folks  with  <5  or  >20  years,  not  much 
in  between) 

■  Not  a  very  deep  bathtub 

■  Minimize  project-  and  personality-dependent  MAJCOM / 
COCOM  coordination 

■  Field  users  drive  most  pre-A  capability  definition  efforts  in  all  four 
domains  (air,  space,  weapons,  C2) 

■  User  community  for  C2  products  is  very  IT-savvy;  things  in  the  IT 
world  tend  to  happen  very  quickly 

■  Immediate  solutions  preferred  over  rigorous  process 

■  Understanding  of  architecture/SoS  concerns 

■  Scope  is  somewhat  dependent  on  domain  (more  significant  for 
space  and  C2,  less  so  for  aircraft  and  weapons) 

■  Frequent  unintended  life  cycle  consequences  of  “IT  now” 

■  On  the  plus  side,  early  SE  is  not  broken  -  our  people 
are  excellent  at  what  they  do 

■  Above  challenges  dilute  effectiveness 
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Program  Success  Factors 

_ (it  ain  't  all  SE’s  fault!) 

□  50%  Politics 


□  40%  Budget 

□  10%Technical  &  Operational  Analysis 

i  “50%  Politics”  translates  to 

“Nothing  happens  without  an 
acceptable  political  compromise” 

i  “40%  Budget”  ...  pretty  much  a  fact  of  life 

■  “10%  Technical  &  Operational  Analysis” 
can  appropriately  inform  (and  influence) 
the  other  90%  of  the  trade  space 

■  If  the  right  team  is  engaging  with 
the  broader  stakeholder  community 

■  If  those  community  members  are 
sufficiently  objective 

■  Concept  Development  personnel/ 
organizations  must  be  politically  astute 

Courtesy  Chris  Leak,  ASC/XRS 
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Way  Forward 


■  Complete  SMC,  ASC  pilot  programs 

■  Socialize  draft  policy/guidebook  throughout 
AF  product  centers  (in  progress) 

■  Develop  forum  for  4  product  center  CD  shops 
to  meet/exchange  ideas,  tools,  personnel 

■  Create  more  stable  funding  environment  for 
CD  efforts 

■  Continuing  working  with  OSD  and  AF 
Requirements  communities  on  incorporating 
early  SE  into  broader  policy 
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Backup 
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Simplifying  &  Scaling  Engineering  Processes: 
Unifying  Business  Units  and  Engineering  Disciplines 

o 


A  New  Order  of  Things 


Rockwell 
Collins 


■  And  it  ought  to  be  remembered  that  there  is  nothing  more  difficult  to  take  in 
hand,  more  perilous  to  conduct,  or  more  uncertain  in  its  success,  than  to 
take  the  lead  in  the  introduction  of  a  new  order  of  things. 

•  Because  the  innovator  has  for  enemies  all  those  who  have  done  well  under 
the  old  conditions,  and  lukewarm  defenders  in  those  who  may  do  well  under 
the  new. 

•  This  coolness  arises  partly  from  fear  of  the  opponents,  who  have  the  laws 
on  their  side,  and  partly  from  the  incredulity  of  men,  who  do  not  readily 
believe  in  new  things  until  they  have  had  a  long  experience  of  them. 

•  Thus  it  happens  that  whenever  those  who  are  hostile  have  the  opportunity 
to  attack  they  do  it  like  partisans,  whilst  the  others  defend  lukewarmly... 

•  Nicolo  Machiavelli,  written  circa  1505,  published  1515 
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Objectives 


Rockwell 
Collins 


■  Understand  basic  &  historical  evolution  of  RCI  engineering 
processes 

•  Understand  steps  through  process  transformation 

•  Explore  outcomes  from  transformation 

•  Explore  expected  resulting  behavior 
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Previous  State  to  2000 


Rockwell 
Collins 


-  Rockwell  Collins  consisted  primarily  of  3  distinct  business  units 

□  Government  Systems 

■  Primary  customers:  military  aerospace 

■  Process  basis:  MIL-STD-499,  DoD-STD-2167 

■  Documented  process:  Engineering  Project  Manual  -  largely  hardware 
development 

□  Business  &  Regional  Systems 

■  Primary  customers:  business  jets,  small  regional  transports,  FAA 

■  Process  basis:  FAA  certification  (RTCA  DO-178) 

■  Documented  process:  Systems  Engineering  Process  -  closely  aligned  with  FAA 
expectations  and  avionics  application 

□  Air  Transport  Systems 

■  Primary  customers:  large  transports,  passenger  &  cargo 

■  Process  basis:  Aircraft  OEM  Processes 

"Documented  process:  Loose  collection  of  process  papers;  OEM-driven  process 
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Reformation 


Rockwell 
Collins 


■  Enterprise  use  of  SAP 

□  New  financial  and  order  administration  capability 

□  How  does  engineering  align  with  SAP? 

•  Cross-  Business  Unit  Functional  Teams 

□  Process  &  Practioner  views  of  new  process 

□  Cross-Business  Teams  Aligned  along  functional  boundaries 

■  Systems  Engineering 

•  Software  Engineering 

■  Hardware  Engineering  (Electrical  &  Mechanical) 

•  ASIC  Engineering 

■  Installation  (latecomer) 
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Consistent  Process  Model:  First  Steps 


Rockwell 
Collins 


■  Technical  Consistent  Process  (TCP)  vl.O  model  released  in  2000 

□  Align  business  units  under  one  process 

□  Align  engineering  process  for  use  with  SAP 


■  TCP  interactive  web-site 

•  TCP  tailoring  worksheet 

•  TCP  computer-based  training 


Rockwell  Collins  Technical  Consistent  Process 


Define 

System,  Manufacturing  and 
Support  Requirements 


- 

Design  1 
System  1 

r 

— r 

Develop  System  Acceptance  i 

Integrate 

Procedures 

[  System 

Perform 

Field/Customer  Support 


ROCKWELL  COLLINS,  INC  PROPRIETARY  INFORMATION 
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Resulting  Behaviors 


Rockwell 
Collins 


Discipline-Centric  views  of  “Engineering” 

□  Inconsistencies  between  disciplines  -  similar,  yet  different 

□  Depth  vs.  breadth  of  process  content 

□  Activity  vs  Phase  Models 
Disparate  Application  of  Process 

□  “Process  Tax” 


Evolving  Process  Library 


Potential  Consistent  Metrics 
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Evolving  the  TCP:  Current  Steps 


Rockwell 
Collins 


■  Design  &  Development  (D&D)  Cycle  Time  Reduction  (CTR) 
Initiative 

□  Simplify  the  engineering  process  model 

•  Eliminate  redundancy 
■  Remove  inconsistencies 

□  Improve  scalability 

□  Improve  user  friendliness  and  information  understanding 

□  Maximize  reuse  of  existing  TCP 


©2007  Rockwell  Collins,  Inc. 


17-Sep-2007 


8 


Observations 


Rockwell 
Collins 


■  Scalability 

□  Consider  TurboTax  approach  to  process  tailoring 
■  Project  &  product  characteristics  drive  process  needs 


□  Allow  common  process  model 

■  Shared  by  business  units 

■  Shared  between  disciplines 
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Transformation 


Rockwell 
Collins 


-  Initial 

□  Cross-  Business  Unit  Functional  Teams 

-  Aligned  along  same  discipline-centric  approach 

■  Allowed  functional  practitioners  voice  to  express  desires  to  reform  TCP 

■  Synthesize  observations  into  ONE  process  model 

-  Subsequent 

□  Mixed  functional  teams 

■  Systems,  software,  hardware,  ASIC  from  different  business  areas 
collaborate 

■  Attempt  to  unify  a  single  engineering  process  model 

■  90+  practitioners  and  process  experts  -  meet  in  smaller  groups  of  10-20 
participants 
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Lean  Engineering:  Simplify  &  Eliminate 


Rockwell 
Collins 


-  Unified  disciplines  under  one  process  model 

□  Eliminate  inconsistencies  between  disciplines 

□  Remove  redundancy 

□  Simplified  process  model 


TCP 

Original 

New 

%  Total 
Reduction 

Activities 

39 

18 

54% 

Tasks 

344 

129 

63% 
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Technical  Consistent  Process  v4.0 


Rockwell 
Collins 


Technical  Management 
Activities  (TMA) 


Technical  Development 
Activities  (TDA) 
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Collins 


a 


Instantiate  Process  Model 


Use  same  process  for: 

•  Primary  end-system 

•  System  components 

•  Enabling  systems  used  to  develop, 
build,  support,  test,  etc  across  the 
primary  system’s  life-cycle. 
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Expected  Behaviors 


Rockwell 
Collins 


■  Process  forms  the  basis  of  project  planning 


Process 

Project  Plan 

( 

\ 

Tailor  Process 
(TurboTCP) 


Project/ 

v 

J 

Product 

Project  WBS 

Characteristics 

•  Consistent  metrics 

•  Consistent  application 
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Conclusion 


Rockwell 
Collins 


■  Understand  basic  &  historical  evolution  of  RCI  engineering 
processes 

□  Business  unit  &  discipline  centric  processes  evolving  to 
unified,  tailorable  engineering  process  model 

•  Understand  steps  through  process  transformation 

□  Collaborative  teams  -  practitioners  and  process  experts 

•  Explore  outcomes  from  transformation 

□  Scalable  process  framework 

•  Explore  expected  resulting  behavior 

□  Process  provides  basis  for  planning  &  execution 
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Integrating  Engineering  Project  Management  and 
Product  Development  Processes 

o 


Objectives 


Rockwell 
Collins 


■  Understand  context  of  RCI  engineering  process  evolution 

•  Understand  roles  &  responsibilities  of  project  managers  within 
RCI  culture 

•  Explore  the  relationship  of  project  management  to  product 
development  processes 

•  Explore  the  roles  of  systems  engineering  vs.  project  management 
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Evolving  Processes:  Current  Steps 


Rockwell 
Collms 


■  Design  &  Development  (D&D)  Cycle  Time  Reduction  (CTR) 
Initiative 

□  Simplify  the  engineering  process  model 

■  Eliminate  redundancy 

■  Remove  inconsistencies 

□  Improve  scalability 

□  Improve  user  friendliness  and  information  understanding 

□  Maximize  reuse  of  existing  engineering  process 

■  Technical  Consistent  Process  (TCP) 
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Problem  Realization 


Rockwell 
Collins 


■  TCP  Primary  Focus:  Engineering  Development 

□  Typical  V-Model  approach  to  engineering 

□  Failed  to  address  Project  Management 

•  How  does  scalability  of  the  process  address  Project  Management 
scalability? 

•  How  should  Project  Management  fit  with  the  engineering  design  & 
development  process? 
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Project  Manager  Definitions 


Rockwell 
Collins 


..  _w also  known  as  Program  Manager  or 

■  Life  Cycle  Value  Stream  Manager  (LCVSM)^J  Produc,  Line  Manager 

□  Life  Cycle  Responsibility  for  prcraucis: - - - 

□  This  includes  business  pursuit  and  capture,  product 
development,  transition  to  production,  customer  delivery  and 
support,  and  transition  out  of  production. 


□  Must  coordinate  activities  across  the 
model;  Engineering,  Manttfsetn 
Service  functions. 


also  known  as  Technical  Director 


ny,  IVL 


■  Technical  Project  Manager  (TPM) 

□  Single  point  of  contact  for  engineering  on  a  development 
project. 

□  Responsible  for  the  technical  leadership  and  project 
management  of  the  design  &  development  activities,  within  the 
guidelines  set  by  the  LCVSM/Program  Manager  and  Customer. 

□  Provides  project  management  expertise  by  planning, 
organizing,  directing,  and  coordinating  functional  department 
activities  to  achieve  cost,  schedule,  and  performance 
requirements 
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Rockwell 
Collins 


Engineering  Leadership:  Common  Purpose,  Different  Roles 


Project  Management 


LCVSM  -«-►  TPM 


Multi-Disciplinary  Team 
Leader 

□  Operations,  Services, 
Finance,  Engineering 

Responsible  for  Profit  &  Loss 


Responsible  for  overall 
project  commitments 

High  Customer  Contact 

□  Business  Development 

□  Enterprise  Coordination 

Covers  project  activities  for 
DP  A  ^  DP  G 


Multi-Disciplinary  Engineering 
Team  Leader 

□  Systems,  Software, 
Electrical,  Mechanical, 
Quality 

Responsible  for  Technical 
Project  Budget 

Responsible  for  ensuring 
project  technical  milestones 
are  satisfied 

High  Customer  Contact 

□  Engineering  Focal  Point 

□  Project  Execution 

Covers  project  activities  for 
DP  C  ^  DP  E 


PE 

Single-Domain  Engineering  Team 
Leader 

□  Either  Single  or  Multi¬ 
discipline 

Responsible  for  WBS  Activity 


Responsible  for  completing 
committed  activities  and  tasks 


Limited  Customer  Contact 
□  Technical  Content 


Covers  project  at  various  Project 
Milestones  (as  needed) 
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Engineering  Leadership  “Work  Allocation” 


Rockwell 
Collins 
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Rockwell 
Collins 


Consistent  Process  Model:  First  Steps 


Technical  Consistent  Process  (TCP)  vl.0  model  released  in  2000 

□  Provides  technical  process  definition 

□  Provides  minimal  project  management  definition 

•  Some  planning  activities 

•  Perform  config  control,  change  control,  peer  reviews,  technical 

r~ 

|  0«I 


reviews 


Rockwell  Collins  Technical  Consistent  Process 
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Project  Management  101 


Rockwell 
Collins 


■  Acquired  Project  Management  course 
□  Fundamentals  of  Project  Management 

•  Convened  project  management  focus  group 

•  Reviewed  Project  Management  Institute  (PMI)  Body  of  Knowledge 


•  Revisited  SAP  Project  Management  model 
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Technical  Consistent  Process  v4.0 


Rockwell 
Collins 


Technical  Management 
Activities  (TMA) 


Technical  Development 
Activities  (TDA) 
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Rockwell 
Collms 


Decisi o n  Pol  nts 


_ Process  Integration 

Management  Authorization  Process 

Note:  Picture  not  in  ten  c/e  d  to  imply  “ti rni ng”,  only  sequence 


Capture  Originating  Requirements 

Define  Operational  Concepts  dWWBlI 

Define  Requirements  bBBB 

Design  Solution 

Implement 
M  Solution  | 

Integrate  y 

Verify  f 
Solution  j 
Validate 

BBBBBBBBBBBIBBBBIMBIBHi  solution 
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I  ime  Progression  (not  to  j 


f 


Support  Solution 
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Project  Management  vs.  Systems  Engineering 


■  System  Engineer  £  Technical  Project  Manager^ 


Can  be  same  person,  but 
roles  are  distinct 


□  Technical  Project  Manager: 


■  Interdisciplinary  role  -  management 

■  Provides  project  management  services  for  design  and 
development  activities  for  a  given  project 

■  Cost,  schedule,  and  project  performance  accountability  to  LCVSM 

■  Work  governed  by  FMA  process;  oversees  overall  TCP  execution 

□  System  Engineer 

■  Interdisciplinary  role  -  technical 

■  Provides  technical  definition  for  a  specific  domain  area  for  a 
project  (technical  domain  expert) 

■  Technical  performance  accountability  to  PE  &  TPM 

■  Work  governed  by  'DA  process;  technical  coordination  between 
disciplines 
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Project  Management  vs.  Systems  Engineering 


Rockwell 

Collins 


System  Engineer  £  Project  Engineer 


Can  be  same  person,  but  roles 
are  distinct 


□  Project  Engineer: 

■  Can  be  from  any  discipline  (system,  software,  hardware,  etc) 

■  Provides  project  management  services  for  a  specific  domain 
and/or  discipline  area  for  a  given  project 

■  Cost,  schedule,  and  project  performance  accountability  to  TPM 

■  Work  governed  by  FMA  process;  oversees  TCP  execution 

□  System  Engineer 

■  Typically  specifically  trained  as  a  system  engineer 

■  Provides  technical  definition  for  a  specific  domain  area  for  a 
project  (technical  domain  expert) 

■  Technical  performance  accountability  to  PE  &  TPM 

■  Work  governed  by  'DA  process;  technical  coordination  between 
disciplines 
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I*  M 


Project  Management  vs.  Systems  Engineering 


Implement  Solution 
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Conclusion 


Rockwell 
Collins 


•  Understand  context  of  RCI  engineering  process  evolution 

□  Needed  to  address  shortcomings  in  project  management 
specific  processes 

•  Understand  roles  &  responsibilities  of  project  managers  within 
RCI  culture 

□  LCVSM,  TPM,  and  PE:  Varying  levels  of  project  management 
responsibility 

•  Explore  the  relationship  of  project  management  to  product 
development  processes 

□  Complimentary  TMA  and  TDA  process  models:  project 
management  and  product  development 

•  Explore  the  roles  of  systems  engineering  vs.  project  management 

□  Complimentary  roles  -  shared  by  some,  distinct  in  others 
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Environment,  Safety,  and  Occupational  Health 
(ESOH)  -  Design  Considerations  to  Support 
Sustainability  and  Readiness 


Ms.  Patricia  Huheey 

Office  of  the  Deputy  Under  Secretary  of  Defense 
Installations  and  Environment 
October  23,  2007 


Outline 


Acquisition,  Technology  and  Logistics 


•  Background 

-  Vision,  Mission,  Goals,  and  Organization 

-  Role  in  Acquisition 

-  Installation  Readiness  Issues  and  Initiatives 

•  Acquisition  ESOH  Policy  and  Guidance 

-  Policy  Objectives  and  Goals 

-  Life  Cycle  ESOH  Risks 

-  DoD  Policy  and  USD(AT&L)  Memorandums 

-  Programmatic  ESOH  Evaluation  (PESHE) 

•  Acquisition  ESOH  Focus  Areas  and  Initiatives 

-  Program  Oversight  and  Reviews 

-  Updates  to  Policy  and  Guidance 

-  Defense  Acquisition  University  (DAU)  Curricula 

-  System  Safety  -  ESOH  Management  Evaluation  Criteria  for  DoD 
Acquisition 

-  “ESOH  in  Acquisition  -  Integrating  ESOH  into  Systems  Engineering” 
Booklet 


Background 


ODUSD(l&E)  Vision  and  Mission 

Acquisition,  Technology  and  Logistics 


Vision:  Installations  assets  and  services  are  available 
when  and  where  needed,  with  the  joint  capabilities 
and  capacities  necessary  to  effectively  and  efficiently 
support  DoD  Missions. 


Mission:  Provide  installations  assets  and  services 
necessary  to  support  our  military  forces  in  a  cost 
effective,  safe,  sustainable  and  environmentally 
sound  manner. 


Background 

ODUSD  (Installations  and  Environment) 


Acquisition,  Technology  and  Logistics 


Integrated 

Management 

Support 

Mr.  Donald  Leonard 


DUSD 

i&tallations  &  Environing 
Mr.  Philip  W.  Grone 


nt' 


Office  of 
Economic 
Adjustment 
Mr.  Patrick  O’Brien 


ADUSD  (Installations) 
Mr.  Chuck  Williams 


ADUSD  (Environment 
Safety  &  Occupational  Heallth) 
Mr.  Alex  Beehler 


Base  Realignment 
&  Closure 
/Ir.  Peter  Potochney 


Business  Enterprise 
Integration 
Ms.  Lora  Muchmore 


Hpusing  &  Competithj 
Sourcing 
Mr.  Joseph  Sikes 


Installations 
Requirements  & 
Management 
Dr.  Get  Moy 


Environmental 

Management 

Ir.  Shah  Choudhury, 
Acting 

Environmental 
Readiness  &  Safety 
Mr.  Curtis  Bowling 

Armed  Forces 
slst  Management  Boali 
Col  Richard  Johnson 


Defense  Explosives 
Safety  Board 
Mr.  Thierry  Chiapellc 


Emerging 
Contaminants 
Is.  Shannon  Cunni 
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Background 


ODUSD(l&E)  Role  in  Acquisition 

Acquisition,  Technology  and  Logistics 

•  AT&L  Advisor  for  ESOH  considerations 

•  Oversight  of  ACAT  ID,  1AM,  and  AT&L  Special  Interest 
programs 

•  Develop/Contribute  ESOH  Input  to  DoDI  5000.2 

•  Identify  OSD  ESOH  expectations  in  the  Defense 
Acquisition  Guidebook  (DAG) 

•  Provide  additional  guidance  for  policy  implementation  on 
the  Acquisition  Community  Connection  (ACC) 

•  Provide  ESOH  input  to  CJCSI/M  3170.01  -  JCIDS 


Background 

ADUSD(ESOH)  Supports  Acquisition 

Acquisition,  Technology  and  Logistics 

•  Chair  DoD  Acquisition  ESOH  IPT 

-  Component  consensus  on  ESOH  in  Acquisition  policy  and 
guidance 

-  Influence  NSS  Acquisition  Policy  and  JCIDS 

-  Opportunity  to  share  tools,  best  practices,  and  collaboration 
across  OSD  and  Components 

•  Member  of  the  Defense  Acquisition  Policy  Working 
Group  (DAPWG) 

-  DoDI  5000.2  and  Defense  Acquisition  Guidebook 

-  ESOH  Special  Interest  Area  on  the  ACC  site 

•  Member  of  Defense  Safety  Oversight  Council  (DSOC) 

-  Co-chair  DSOC  Integration  Group 

-  Acquisition  and  Technology  Programs  Task  Force 


•  Realistic  training  requires  realistic 
training  environments 


•  Ability  to  field  and  use  advanced 
military  technology  is  fundamental 
to  U.S.  warfare 


•  Live  Fire  is  fundamental  to  training 


Background 

Training  and  Testing  Are  Important  As  Ever 

Acquisition,  Technology  and  Logistics 


•  Our  weapons  and  tactics  require 
increasingly  large  battlespaces 


•  Readiness  is  perishable  -  Skills 
must  be  maintained  through 
regular  training 

•  OPTEMPO,  PERSTEMPO  are  up 
-  ready  access  to  training  is 
essential 


We  must  train  as  we  fight 
because  we  fight  as  we  train 


Background 


Bases  and  Ranges  Are  at  Risk 

Acquisition,  Technology  and  Logistics 


Encroachment:  Restrictions  that  inhibit  accomplishment  of  our  live  training 

and  testing  as  required 


•  Force  Readiness  is  fundamentally  linked  to  the  quality  and  frequency  of  test  and 
training 

•  The  impact  of  encroachment  Is  broad  --  affecting  our  ability  to  execute  realistic 
air,  ground,  and  naval  training  across  the  nation,  as  well  as  beyond  its  borders 


Background 


Incompatible  Development.... 

Acquisition,  Technology  and  Logistics 


...Inhibits  Effective 
Military  Training 


•  Increases  complaints 
of  noise,  dust  and 
other  training  related 
activities  from  new 
neighbors 

•  Reduces  or 
eliminates  any  future 
mission  flexibility 

•  Limits  off-range 
species  and  habitat 
conservation 
opportunities 


Fort  Bragg,  NC 
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Background 


Sustainable  Ranges  Initiative 

Acquisition,  Technology  and  Logistics 


•  Sustainable  Ranges  are  the 
Foundation  for  Future  Training 

-  Readiness  Depends  on  Range 
Access  and  Live  Fire  Capability 

-  Sound  Range  Stewardship 
Supports  Training  Realism  and 
Environmental  Excellence 

-  Working  “Outside  the  Fence”  is  a 
Must 

•  Requires  a  Comprehensive 
Solution  That  Integrates 

-  Training  Requirements 

-  Natural  Environment  & 
Stewardship 

-  Community  Economic  and  Social 
Interests 

-  Legislative  and  Regulatory 
Framework 


Range  Sustainment  Initiative 
Strategy  Elements 

•  Policy 

•  Programming 

•  Leadership  & 
Organization 

•  Legislation  &  Regulation 

•  Outreach  and 
Engagement 

•  Compatible  Land  Use  & 
Buffering 

•  Comprehensive 
Reporting 
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Background 

Natural  Infrastructure  Management  (NIM) 

Acquisition,  Technology  and  Logistics 


•  To  be  mission  ready  at  the  installation  and  range  level, 
in  addition  to  built  infrastructure  and  personnel,  DoD 
needs  an  adequate  supply  of  air,  land,  and  water 
assets  (i.e.,  natural  infrastructure)  to  train,  test,  and 
perform  its  varied  missions 

•  The  Natural  Infrastructure  Capability  Work  Group 
(NICWG)  is  working  to  develop  and  implement  a 
common  framework  for  NIM  to  support  mission 
requirements 
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NIM  Framework 


Background 


Acquisition,  Technology  and  Logistics 


•  NICWG  tested  and 
refined  two  major  aspects 
of  a  common 
management  framework: 

-  Natural  Infrastructure 
assessments  provide  data 
to  determine  Nl  “readiness” 
to  meet  current  and 
emerging  operational 
requirements 

-  Natural  Infrastructure  asset 
valuation  is  used  to  assess 
military,  ecological,  and 
economic  values  provided 
by  Nl  assets 


Acquisition  programs  and 
operators  must  provide  system 
specific  data  to  support  NIM 


Acquisition,  Technology  and  Logistics 


Acquisition  ESOH  Policy  and  Guidance 
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Policy  and  Guidance 

Plan  Ahead  &  Influence  Design 

Acquisition,  Technology  and  Logistics 

•  Identify  system  life  cycle  ESOH  risks  early  to  influence 
design,  not  address  them  afterwards  as  operational 
considerations 

•  System  design  is  most  effectively  influenced  through  the 
system  engineering  (SE)  process 

-  Active  participation  in  the  IPTs  is  critical  to  success 

•  ESOH  hazards  and  associated  risks  are  best  managed 
using  a  standard  approach  and  structured  process 

•  E,  S,  and  OH  inputs  to  SE  must  be  optimized  across  the 
disciplines  to  meet  cost  and  performance  requirements 
and  needs  throughout  the  life  cycle 
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Policy  and  Guidance 


Life  Cycle  ESOH  Risks 

Acquisition,  Technology  and  Logistics 

•  ESOH  risks  may  include 

-  Toxic  and  hazardous  materials  and  wastes 

-  Environmental  and  occupational  noise 

-  Impacts  to  personnel  safety  and  occupational  health 

-  Inability  to  maintain  regulatory  compliance  requirements 

•  Need  to  manage  ESOH  risks  associated  with 

-  Routine  system  development,  testing,  training,  operation, 
sustainment,  maintenance,  and  demilitarization/disposal 

-  System  failures  or  mishaps,  including  critical  software  failures 

-  Life  cycle  cost,  schedule,  and  performance  impacts  from  ESOH 
compliance  requirements 
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Policy  and  Guidance 

Top  Level  DoD  Acquisition  ESOH  Policy 

Acquisition,  Technology  and  Logistics 

•  Use  a  total  systems  approach  to  minimize  or  eliminate 
characteristics  that  produce  environmental,  safety  or 
occupational  health  hazards,  where  practicable  and  cost 
effective 

•  Address  safety  throughout  the  acquisition  process 

•  Use  the  system  safety  methodology  in  MIL-STD-882D  to 
eliminate  ESOH  hazards  where  possible  and  manage  ESOH 
risks  where  hazards  cannot  be  eliminated 

•  Ensure  formal  risk  acceptance  at  designated  management  level 

•  Document  hazardous  materials  associated  with  the  system  and 
plan  for  their  safe  disposal,  beginning  during  system  design 

•  Provide  safety  releases  to  developmental  and  operational 
testers  prior  to  any  test  using  personnel 


16 


Policy  and  Guidance 

Recent  USD(AT&L)  Policy  Memorandums 

Acquisition,  Technology  and  Logistics 

•  Defense  Acquisition  System  Safety,  23  September  2004 

-  Use  Standard  Practice  for  System  Safety,  MIL-STD-882D  to  manage 
ESOH  risk 

-  Report  ESOH  risk  status  and  acceptance  decisions  at  technical  and 
program  reviews 

•  Reducing  Preventable  Accidents,  21  November  2006 

-  Develop  process  to  provide  ESOH  input  for  JCIDS 

-  Address  the  status  of  each  High  and  Serious  ESOH  risk  and  compliance 
with  applicable  safety  technology  requirements  at  all  program  reviews 

-  PMs  will  support  system-related  Class  A  and  B  mishap  investigations 

•  Defense  Acquisition  System  Safety  -  ESOH  Risk  Acceptance, 

7  March  2007 

-  Formal  acceptance  of  ESOH  risks  prior  to  exposing  people,  equipment,  or 
the  environment  to  a  known  system-related  ESOH  hazard 

-  User  representative  formal  concurrence  for  Serious  and  High  ESOH  risks 


Policy  and  Guidance 

Programmatic  ESOH  Evaluation  (PESHE) 

Acquisition,  Technology  and  Logistics 

•  PESHE  document  communicates  ESOH  planning  and  the 
status  of  ESOH  risk  management  for  the  system 

•  PESHE  must  include: 

-  Identification  of  ESOH  responsibilities 

-  The  strategy  for  integrating  ESOH  considerations  into  the  systems 
engineering  process 

-  Identification  of  ESOH  risks  and  their  status 

-  A  description  of  the  method  for  tracking  hazards  throughout  the  life 
cycle  of  the  system 

-  A  compliance  schedule  for  National  Environmental  Policy  Act  (NEPA) 
and  Executive  Order  12114 

•  All  programs,  regardless  of  ACAT,  are  required  to  prepare  a 
PESHE  to  support  Program  Initiation  for  Ships,  MS  B,  MS  C, 
and  Full-Rate  Production  Decision  Review 
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Acquisition,  Technology  and  Logistics 


Acquisition  ESOH  Focus  Areas  and  Initiatives 


Focus  Areas  and  Initiatives 

ACAT  I  Program  Oversight/Reviews 

Acquisition,  Technology  and  Logistics 

•  Participate  in  MS  review  process 

-  Attend  program  WIPT/OIPT  meetings 

-  OSD  coordination/review  of  program  documents 

o  Acquisition  Strategy 
o  PESHE 

-  Assist  Component  and  program  staff  to 

o  Clarify  DoDI  5000.2  ESOH  policy  requirements 
o  Emphasize  integration  of  ESOH  into  SE 
o  Focus  on  life  cycle  ESOH  risk  management 


•  Developing  a  pilot  to  identify  procedures  to  implement 
briefing  ESOH  risks  at  acquisition  program  reviews 

-  Defining  expectations  of  PM  reporting  is  critical 
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Focus  Areas  and  Initiatives 


•  Input  to  DoDI  5000.2  (in  SD  106  coordination) 

-  Updated  and  moved  ESOH  requirements  to  the  new  Enclosure  12  - 
Systems  Engineering 

-  Incorporated  the  3  USD(AT&L)  memorandums 

-  Clarified  existing  DoDI  5000.2  policy  requirements 

-  Hazardous  materials,  wastes,  and  pollutants  (discharges/emissions/ 
noise)  associated  with  the  system  must  be  included  in  the  PESHE 

-  PM  shall  provide  system-specific  analyses  and  data  to  support  other 
organizations’  NEPA  and  EO  121 14  analyses 

•  Updated  ESOH  input  for  the  Defense  Acquisition  Guidebook 

-  “System  Safety  Analyses”  added  to  the  SE  V  model  Inputs  and  Outputs 
and  to  “The  Wall  Chart” 

-  Clarified  expectations 

•  Initiating  update  for  ESOH  Special  Interest  Area  on  the  ACC 
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Focus  Areas  and  Initiatives 

System  Safety  -  ESOH  Management 

Evaluation  Criteria 

Acquisition,  Technology  and  Logistics 

•  Tool  to  assess  an  acquisition  program’s  overall  management  of 
ESOH  as  part  of  SE  process 

-  Technical  and  Program  Reviews  (self  assessment) 

-  Milestone  Review  Process  (oversight  assessment) 

•  Four  key  areas  for  evaluation 

-  Planning 

-  Requirements  Analysis 

-  Hazard  Analysis 

-  Resources 

•  Assessment  criteria  for  key  each  area  for  each  life  cycle  phase,  and 
can  be  combined  for  overall  rating  for  each  life  cycle  phase 

•  To  be  incorporated  into  Defense  Acquisition  Program  Support 
(DAPS)  SE  Assessment  Methodology 

•  Available  on  ACC  at  https://acc.dau.mil/esoh  and  included  in  booklet 
handouts 
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Focus  Areas  and  Initiatives 


DAU  Curricula 

Acquisition,  Technology  and  Logistics 

•  Working  with  DAU  to  improve  ESOH  training  for  the 
acquisition  workforce  since  2000 

•  Review  courses  and  provide  ESOH  input  consistent  with 
current  DoD  policy,  guidance,  and  best  practices 

-  Courses  reviewed  include  Acquisition;  Systems 

Engineering;  Program  Management;  Logistics;  Test  and 
Evaluation;  Software  Acquisition;  Facilities  Engineering; 
and  Production,  Quality,  and  Manufacturing 

•  Technical  Management  Functional  Advisor  support  and 
DSOC  funding  provided  means  to  more  aggressively  review 
and  provide  ESOH  input  for  the  courses  over  past  2  years 
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Focus  Areas  and  Initiatives 

DAU  CLE009  -  System  Safety  in  SE 

Acquisition,  Technology  and  Logistics 


•  ODUSD(l&E)  and  ODUSD(A&T)/SSE  effort 

•  Roadmap  for  using  the  MIL-STD-882D  system  safety 
methodology  for  eliminating  or  mitigating  ESOH  hazards 
and  associated  risks 

•  Discusses  system  safety/ESOH  efforts  to  be  conducted 
throughout  the  system’s  life  cycle 

•  Helps  define  how  ESOH  and  SE  communities  should  work 
together 

•  Available  on-line  since  April  2005  at  https://learn.dau.mil 

•  Over  1 600  graduates  to  date 
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Focus  Areas  and  Initiatives 


Integrating  ESOH  Into  SE  Booklet 

Acquisition,  Technology  and  Logistics 


Builds  on  CLE009 
and  depicts  when 
ESOH  activities 
should  be  performed 
to  influence  system 
design  throughout 
the  systems 
engineering  process 


System  Safety-ESOH 
Management 
Evaluation  Criteria 
are  included 


Copies  available  here,  and  will  be  discussed  during  System  Safety  Track, 
Session  4B7  on  Thursday,  25  Oct  10:15am-12pm 
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Focus  Areas  and  Initiatives 

Way  Ahead 

Acquisition,  Technology  and  Logistics 

•  Continue  to  mature  the  ESOH  risk  management  process 
for  acquisition 

-  Update  guidance  on  PESHE  requirements  and 
expectations 

-  Revise  MIL-STD-882D  to  better  support  DoD  policy  and 
initiatives,  expand  importance  of  integration  into  SE,  and 
inclusion  of  environmental  efforts 

-  Provide  guidance  on  implementing  ESOH  risk  acceptance 
policy 

-  Develop  tools  to  help  identify  system  ESOH  requirements 

•  Continue  to  lead  and  support  ESOH  efforts  to  influence 
system  design  and  development 
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Outline 


■  System  Assurance  Defined 

■  The  System  Assurance  Problem  Space 

■  Software  As  A  Root  Cause  Problem 

■  Engineering  Shortfalls 

■  Seven  Systems  Engineering  Community 
Leadership  Challenges 

■  Guidance  For  Systems  Assurance 

■  Standardization  In  Support  Of  Systems 
Assurance 

■  Summary 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 


System  Assurance  Defined 


System  assurance  is  the  level  of 
confidence  that  the  system 
functions  as  intended  and  is  free 
of  exploitable  vulnerabilities, 
either  intentionally  or 
unintentionally  designed  or 
inserted  as  part  of  the  system. 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 


System  Assurance  Problem 

Space 


■  Large-scale  systems  and  systems  of  systems  represent  a 
complex  supply  chain  integrating 

-  Proprietary  and  open-source  software 

-  Legacy  systems 

-  Hardware 

-  Firmware 

■  These  systems  are  sourced  from  multiple  suppliers  who  employ 
people  from  around  the  world 

■  Most  systems  we  encounter  today  contain  software  elements 
and  most  depend  upon  software  for  a  good  portion  of  their 
functionality 

■  Technologies  to  build  reliable  and  secure  software  are 
inadequate 

-  Our  ability  to  develop  software  has  not  kept  pace  with  hardware 
advances 

-  Can’t  construct  complex  software-intensive  systems  for  which  we 
can  anticipate  performance 

■  Assurance  is  a  full  life  cycle  systems-level  problem 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 


Software  As  A  Root  Cause 

Problem 


■  System  risk  has  dramatically  increased  due  to  the 
simultaneous  growth  in  software  vulnerabilities  and 
in  threat  opportunities 

■  Risk  management  processes  inadequately  address 
these  threats  and  risks 

■  Threats  presented  by  suppliers  of  software  products 
and  services  are  not  adequately  identified  and 
analyzed 

i  Development  and  acquisition  processes 
inadequately  address  software  security 

■  There  is  a  fundamental  lack  of  both  the  scientific 
understanding  of  software  risks  and  the  capabilities 
to  effectively  diagnose  and  mitigate  in  the  in  a  timely 
manner 

Source:  J.  Jarzombek.  DOD  Software  Assurance 

Initiative:  Mitigating  Risks  Attributable  to  Software. 

DOD  Software  Assurance  Forum,  July  2004. 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 


Or,  More  Succinctly  .  .  . 


■  There  is  a  failure  to  assure  correct, 
predictable,  safe,  secure  execution  of 
complex  software  in  distributed 
environments 

■  Inadequate  attention  is  given  to  the  total 
lifecycle  issues,  including  impacts  on 
lifecycle  cost  and  risk  associated  with  the 
use  of  commercial  or  reused  products  and 
components 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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System  Assurance 
Engineering  Shortfalls 


■  Current  techniques  for  specifying,  building, 
demonstrating,  and  verifying  assured  components 
with  well  understood  properties  are  not  cost-effective 
or  scaleable 

■  Cannot  easily  infer  the  assurance  properties  of  a 
system,  or  systems  of  systems,  from  component 
level  assurance  information 

■  Don’t  know  enough  about  composability  problems 
and  emergent  behavior  when  components  are 
interconnected  in  large-scale  systems  and  systems 
of  systems 

■  Exhaustive  testing  to  rule  out  vulnerabilities  is 
generally  not  feasible  due  to  the  size  and  complexity 
of  our  systems  of  interest 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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The  Systems  Engineering 

Challenge 


Integrating  a  heterogeneous  set  of  globally 
engineered  and  supplied  proprietary,  open- 
source,  and  other  software;  hardware;  and 
firmware;  as  well  as  legacy  systems;  to 
create  well-engineered  integrated, 
interoperable,  and  extendable  systems 
whose  security,  safety,  and  other  risks  are 
acceptable  -  or  at  least  tolerable. 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Systems  Engineering  Community 
Leadership  Challenges  #1  - 

Acquisition 

Collaboration  to  develop  new 
approaches  and  improve  existing 
approaches,  standards,  and  tools 
that  address  systems  assurance 
issues  throughout  the  acquisition  life 
cycle  and  the  supply  chain 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Systems  Engineering  Community 
Leadership  Challenges  #2  - 
Engineering  Practices 

Integration  of  systems  and  software 
engineering  practices  for  producing 
system  architectures  and  resulting 
systems  that  are  resistant  to  intrusion 
and  compromise 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Systems  Engineering  Community 
Leadership  Challenges  #3  - 

Research 

Sponsor  research  into  new 
modalities  for  system  composition  to 
meet  specific  assurance  objectives 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Systems  Engineering  Community 
Leadership  Challenges  #4  - 
Quality  Attributes 

Define  systems  and  software 
assurance  quality  attributes  that  can 
be  addressed  during  architectural 
tradeoffs 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Systems  Engineering  Community 
Leadership  Challenges  #5  - 
Standardization 

Encourage  the  development  of 
commercial  standards  addressing 
vulnerability  management  throughout 
the  supply  chain,  including  product- 
level  and  component-level 
specifications  and  standards  for 
detecting  component  vulnerabilities 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Systems  Engineering  Community 
Leadership  Challenges  #6  - 
Policy  and  Guidance 

Develop  policy,  guidance,  and 
training  for  the  acquisition  of  systems 
with  desired  assurance  properties 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Systems  Engineering  Community 
Leadership  Challenges  #7  - 
Life  Cycle  Planning 

Ensure  that  life  cycle  issues  and 
tradeoffs  associated  with  the 
incorporation  of  commercial 
components  and  reused  software 
into  systems  are  clearly  addressed  in 
program  plans,  systems  engineering 
plans,  test  and  evaluations  plans, 
and  during  periodic  reviews 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 

10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Guidance  For  Systems 
Assurance  - 1 


■  Systems  Assurance  -  Delivering 
Mission  Success  in  the  Face  of 
Developing  Threats 

-An  NDIA  guidebook  intended  to 
supplement  the  knowledge  of  systems 
(and  software)  engineers  who  have 
responsibility  for  systems  for  which 
there  are  assurance  concerns 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 


NDIA/DoD  System  Assurance 
Guidebook  -  Scope 


■  Practical  guidance  for  the  Government  acquisition 
community,  industry,  academia,  and  other 
commercial  and  government  partners 

■  Synthesis  of  knowledge  gained  from  existing 
practice,  recommendations,  policy,  and  mandate, 
rather  than  reinventing  the  wheel 

■  Recap  of  important  concepts  and  principles  from 
foundational  documents,  standards,  and  mandates, 
and  discussion  of  them  in  the  larger  context  of 
systems  assurance  as  presented  by  the  White  Paper 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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NDIA/DoD  System  Assurance  Guidebook  - 
Mapped  To  ISO/IEC/IEEE  15288 


■  Agreement  Processes 

-  Acquisition 

-  Supply 

■  Project  Processes 

-  Project  Planning 

-  Project  Assessment 

-  Project  Control 

-  Decision-making 

-  Risk  Management 

-  Configuration  Management 

-  Information  Management 

■  Assurance  Case  Process 


■  Technical  Processes 

-  Stakeholder  Requirements 
Definition 

-  Requirements  Analysis 

-  Architectural  Design 

-  Implementation 

-  Integration 

-  Verification 

-  Transition 

-  Validation 

-  Operation 

-  Maintenance 

-  Disposal 


Enterprise  Processes 

-  Enterprise  Environment 
Management 

-  Investment  Management 


System  Life  Cycle  Process 
Management 

Resource  Management  [including 
human  resource  training] 

Quality  Management 


10th  Annual  NDIA  Systems  Conference,  23  October  2007,  Track  7,  3:45  PM 
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Alignment  of  Standards  In  The  Guidebook 


Concept 

Stakeholder  Heeds 

Explore  Concepts 

Propose  liable  Solutions 

Development 

Refine  System  Requirements 

Production 

Utilization 

Operate  System  to 

Satis^- User  Heeds 

Retirement 

Create  Solution  Description 

Build  System 

Verify  ft  Vdidale 

Produce  Systems 

Inspect  &  Test 

Support 

Provide  Sustained 
System  Capability 

Store,  Archive,  or 

Dispose  ofthe  System 

ISO/IEC  15288 :2002(E) 


IEEE  1220-2005 


Concept 

System 

Definition 

Preliminary 

Design 

Detailed 

Design 

FAIT* 

Production 

Utilization 

Support 

Retirement 

Support 

*  Fabrication,  Ass^mbty', 
Integration,  &  T[est 


Integrated  Defense  Acquisition,  Technology,  &  Logistics  Life  Cycle  Framework 

i  ME  A  MSB  i  MS  C 

_ A _ | _ A. 


JCIDS 

Heeds: 

FAA 

FHA 

ESA 


Concept 

Decision 


DCR  ICD 


i  (HR-KPP) 

i 

Defense 

Acquisition 

System 


Concept 

Refinement 


Technology 

Development 

HR-KPP 


muaJ 


(EP) 


iU, 

SEP 


(EP) 


t  CDDd 

-Pre- System! Acquisition  ■ 


seJp 

IDA 

■  T.  1 


System  Development  & 
Demonstration 
I  HR-KPP 


EP 


CDD 
- 


Systems  /  Systems  SIpP, 

Integration  /  Demonstration  IC 

System  Functional  cpjj 

- System  A  c  quisition 


IOC  FOC  i 

j^AAA 


Production 

& 

Deployment 


Operations 

& 

Support 


AISP 

LRIP 


Final  Product] 
Eveline 


(EP) 

Sustainment^1 

Sustainment  - 


Uqefd 


Initiation 

A  c  quisition/D  e  velopment 

I  mple  mentation/A  ss  essment 

Operations  Maintenance 

Disposition 

1  -  Business  Partner  Engagement 

1  -  Risk  Assessment 

1  -Pro  dict/Ccmponeit  Inspection  ft 

1  -  Change  Control  ft  Auditing 

1  -  Thins  ition  Flaming 

2  -  Document  Enterprise  Architecture 

2  -  Initial  Security  Baseline  Coitrols 

Acceptance 

2  -  Continuous  Mbnioring 

2  -  Component  Disposal 

3  -  Identify/Specific  Applicable  Policies  ft  Laws 

3  -  Refinement  of  S  ecurity  Baseline 

2  -  Security  Control  Megrarizm 

3  -Re-Certification 

3  -  Media  Sanitation 

4  -  Develop  C,L&  A  Objectives 

Controls 

3  -User/Administrative  Guidance 

4  -Re -Accreditation 

4  -  Information  Archiving 

f  -  Information  ft  Information  System  Security 

4  -  Security  Control  Baseline 

4  -  System  Security  Test  ft  Evaluation 

5  -EncidentHandling 

Categorisation 

5  -  Cost  Analysis  ft  Reporting 

Plan 

0  -  Audiing 

6  -  Process  Specification  Development 

-  Security  Planning 

5  -  Security  Certification 

7  -  Intrusion  Detection  ft  Monitoring 

7  -  Preliminary  Risk  Assessment 

7  -  Unitflhte  Ration  Security  Test  ft 

6  -  Statement  of  Residual  Risk 

S  -  Contingency  Planning  (including  Continuity  of 

Evaluation 

7  -Security  Accreditation 

Operations  Plan  [COOP]) 

NIST  Information  Security  and  the  System  Development  Life  Cycle 
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Guidance  For  Systems 
Assurance  -  2 


■  State  of  the  Art  Report  on  Software 
Security  Assurance 

-  An  IATAC/DACS  report  identifying  and 
describing  the  current  state  of  the  art  in 
software  security  assurance,  including  trends 
in: 

■  Techniques  for  the  production  of  secure  software 

■  Technologies  that  exist  or  are  emerging  to  address 
the  software  security  challenge 

■  Current  activities  and  organizations  in  government, 
industry,  and  academia,  in  the  U.S.  and  abroad, 
that  are  devoted  to  systematic  improvement  of 
software  security 

■  Research  trends  worldwide  that  might  improve  the 
state  of  the  art  for  software  security 
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Guidance  For  Systems 
Assurance  -  3 


■  Secure  Software  Assurance:  A 
Guide  to  the  Common  Body  of 
Knowledge  to  Produce,  Acquire, 
and  Sustain  Secure  Software 

-A  DHS  guidebook  intended  as  a 
framework  to  identify  workforce  needs 
for  competencies  and  leverage 
standards  and  best  practices  to  guide 
software-related  curriculum 
development 
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Guidance  For  Systems 
Assurance  -  4 


■  Security  in  the  Software  Life  Cycle: 
Making  Software  Development 
Processes  -  and  the  Software  Produced 
by  Them  -  More  Secure 

-  An  DHS  report  providing  a  compendium  of 
methodologies,  life  cycle  process  models, 
sound  practices,  and  supporting  technologies 
that  would,  if  adhered  to,  increase  software 
security 
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Guidance  For  Systems 
Assurance  -  5 


■  Software  Assurance  in 
Acquisition:  Mitigating  Risks  to 
the  Enterprise 

-  A  DHS  report  intended  to  provide 
guidance  on  enhancing  supply  chain 
management  through  improved  risk 
mitigation  and  contracting  for  secure 
software 
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Standardization  In  Support  Of 
Systems  Assurance  -  Languages 


■  ISO/IEC  SC22  (Languages) 

-  ISO/IEC  Technical  Report  -  Guidance 
for  Avoiding  Vulnerabilities  through 
Language  Selection  and  Use 

■  Comparative  guidance  spanning  multiple 
programming  languages 

■  Goal:  Avoidance  of  programming  errors 
that  lead  to  vulnerabilities 
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Standardization  In  Support  Of 
Systems  Assurance  -  Security 


■  ISO/IEC  SC27  (IT  Security 
Techniques) 

-  ISO/IEC  21827,  System  Security 
Engineering  Capability  Maturity  Model 
(SSE  CMM) 

-  ISO/IEC  15443  (FRITSA),  A 
framework  for  IT  security  assurance 

-  ISO/IEC  DTR  19791,  Assessment  of 
Operational  Systems 
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Standardization  In  Support  Of 
Systems  Assurance  -  Safety 


■  IEC  SC  65A  (Functional  Safety) 

-  IEC  61508,  Functional  Safety 

■  Risk-based  approach  for  determining  the  required 
performance  of  safety-related  systems 

i  Requirements  based  on  common  underlying 
principles  to  facilitate: 

-  Supply  chain  efficiencies 

-  Clear  communication  of  requirements 

-  Development  of  techniques  and  measures 

-  Development  of  conformity  assessment  models 
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Standardization  In  Support  Of 
Systems  Assurance  -  System 
and  Software  Assurance 


24748:  Guide  to  Life  Cycle  Management 


Other 
standards 
providing 
details  of 
selected  SW 
processes 


Source:  J.  Moore,  SC7 
Liaison  Report,  IEEE 
Software  and  Systems 
Engineering  Standards 
Committee,  Executive 
Committee  Winter 
Plenary  Meeting, 
February  2007. 


Revised  12207: 

Life  cycle 
processes  for 
SW 


Revised 

15289: 

Document¬ 

ation 


Revised  15288: 

Life  cycle 
processes  for 
systems 


Interoperation 


A 

V/ 


Revised 

16326: 

Project 

Mcjmt 


Revised 

15939: 

Measure¬ 

ment 


Revised 

16085: 

Risk 

Mgmt 


A 

V/ 


Other 
standards 
providing 
details  of 
selected 
system 
jDrocesses_ 


15026: 
Additional 
practices  for 
higher 
assurance 
systems 


A 

V/ 


Common  vocabulary,  process  architecture,  and  process  description  conventions 


ISO/IEC/IEEE  15026,  System  and  Software  Assurance 
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ISO/IEC/IEEE  15026,  System 
and  Software  Assurance 


■  A  65-page  draft  has  been  balloted 

-  It  incorporates  material  from  FCD  12207,  FCD  15288, 
ISO/IEC  15289,  IEEE  Std  1228  and  the  safety  and 
security  extensions  to  CMMI 


The  draft  contains  requirements  and  guidance 
for: 


-  Assurance  cases 

-  Associated  documents,  e.g.  assurance  plan,  reports, 
analyses 

■  The  process  view  is  comprehensive — it  touches 
every  process  of  15288  and  12207  (except  for 
the  enterprise  processes) 

■  Joint  comment  resolution  is  in  progress 


Source:  J.  Moore,  Proposed  Revision  of  ISO/IEC  15026:  Status  Report, 
IEEE  Software  and  Systems  Engineering  Standards  Committee,  Executive 
Committee  Summer  Plenary  Meeting,  July  2007. 
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Current  Draft  of  Scope  Clause 


■  This  International  Standard  provides  requirements  for  the 
life  cycle  including  development,  operation,  maintenance, 
and  disposal  of  systems  and  software  products  that  are 
critically  required  to  exhibit  and  be  shown  to  possess 
properties  related  to  safety,  security,  dependability,  or 
other  characteristics 

■  It  defines  an  assurance  case  as  the  central  artefact  for 
planning,  monitoring,  achieving  and  showing  the 
achievement  and  sustainment  of  the  properties  and  for 
related  support  of  other  decision  making 

■  The  interaction  of  the  requirements  for  the  assurance 
case  with  life  cycle  processes  implies  a  normative 
interpretation  of  the  processes  from  ISO/IEC  15288 
and  ISO/IEC  12207 

■  Finally,  the  standard  provides  requirements,  in  addition  to 
those  of  ISO/IEC  15289,  for  information  artefacts  that 
result  from  those  processes 

Source:  J.  Moore,  Proposed  Revision  of  ISO/IEC  15026:  Status  Report,  IEEE  Software  and  Systems  Engineering 

Standards  Committee,  Executive  Committee  Summer  Plenary  Meeting,  July  2007. 
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Relationships  to  Other 
Standards 


■  The  provisions  regarding  process  in  this  international  standard  make 
extensive  normative  references  to  ISO/IEC  12207:2007  and  ISO/IEC 
15288:2007,  the  international  standards  for  software  and  system  life 
cycle  processes. 

■  Users  of  this  international  standard  will  probably  require  risk 
management  and  measurement  processes  that  are  more  fully  detailed 
than  the  treatment  provided  in  ISO/IEC  15288.  Two  international 
standards,  ISO/IEC  16085  and  ISO/IEC  15939  are  useful  in  this  regard. 

■  The  provisions  regarding  the  assurance  plan  and  assurance  case  are 
intended  to  be  compatible  with  the  provisions  of  ISO/IEC  15289:2006  for 
information  items  resulting  from  life  cycle  processes. 

■  Some  material  regarding  assurance  planning  and  its  supporting  analyses 
has  been  adapted  from  IEEE  Std  1228:1994. 

■  The  provisions  regarding  product  characteristics  are  intended  to  be 
generally  consistent  with  those  of  the  ISO/IEC  25000  series  of  standards 
related  to  product  quality,  the  ISO/IEC  27000  series  of  standards  related 
to  information  security  management  systems,  the  IEC  61508  standard 
on  functional  safety,  and  various  standards  of  IEC  TC  56  related  to 
dependability. 

■  ISO/IEC  TR  15443,  Information  technology-Security  techniques--A 
framework  for  IT  security  assurance,  discusses  the  need  for  arguments 
and  evidence  in  the  IT  context. 
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Current  Draft  Conformance 

Clause 


This  international  standard  is  intended  to  be  used  in  conjunction 
with  ISO/IEC  12207  and  ISO/IEC  15288.  This  standard  provides 
requirements  and  guidance  in  addition  to  that  of  the  referenced 
standards.  To  conform  to  this  international  standard,  one  conforms 
to  the  referenced  standards  and  conforms  to  the  additional 
requirements  of  this  international  standard. 

It  is  permitted  to  assert  conformance  for  specified  properties  of 
the  system  or  to  specified  portions  of  the  system  if  the  assertion 
is  accompanied  by  a  clear  statement  of  the  limitations  ... 

One  may  assert  conformance  to  this  standard  for  specific 
product  claims  related  to  stated  critical  properties  or 
characteristics  for  specifically  identified  versions  of  products 
or  portions  of  products  under  specified  conditions. ... 

Assertions  of  conformance  that  relate  only  to  the  lack  of  limited 
categories  of  faults  or  weaknesses  must  not  be  stated  as  claims  for 
more  general  properties  such  as  security  or  safety. ... 

Source:  J.  Moore,  Proposed  Revision  of  ISO/IEC  15026:  Status  Report, 

IEEE  Software  and  Systems  Engineering  Standards  Committee,  Executive 
Committee  Summer  Plenary  Meeting,  July  2007. 
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General  Requirements  on 
Assurance  Cases 


i  The  project  shall  establish  and  maintain  an  assurance  case. 

■  The  project  shall  ensure  that: 

-  Goals  and  objectives  for  safety,  security,  dependability  and  any  other  designated  critical 
properties  are  formulated. 

-  Product  assurance-related  objectives,  properties,  or  characteristics  are  explicitly  selected  for 
special  attention  and  application  of  this  standard  to  address  the  goals  and  objectives. 

-  Requirements  for  the  achievement  of  these  objectives,  properties,  or  characteristics  are 
defined. 

-  Measures  for  the  requirements  are  selected  and  related  to  the  desired  characteristics. 

-  Criteria  for  the  achievement  or  degree  or  achievement  of  these  objectives,  properties,  or 
characteristics  are  selected  and  traced  to  requirements. 

-  Approaches  for  achieving  the  objectives,  properties,  or  characteristics  are  planned, 
designed,  and  implemented,  as  well  as  demonstrating  and  documenting  that  achievement. 

-  The  extent  of  achievement  is  continuously  monitored,  documented,  and  communicated  to 
stakeholders  and  managers. 

-  An  assurance  case  documenting  and  communicating  the  extent  of  achievement  is  specified, 
developed,  and  maintained  as  an  element  of  the  system. 

-  The  artefacts  for  documenting,  analyzing,  and  communicating  the  required  or  claimed 
properties  and  characteristics  and  the  extent  of  achievement  are  specified,  developed,  and 
maintained. 


Requirements  of  the  approval  authority  are  satisfied  and  necessary  licenses  or  certifications 
are  received. 

Source:  J.  Moore,  Proposed  Revision  of  I  SO/I  EC  15026:  Status  Report,  IEEE  Software  and  Systems 
Engineering  Standards  Committee,  Executive  Committee  Summer  Plenary  Meeting,  July  2007. 
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The  Assurance  Case  In  Relation  To 

The  Product  And  Its 
Quality/Assurance  Factors 


Attributes 

□  Clear 

□  Consistent 

□  Complete 

□  Comprehensible 

□  Defensible 

□  Bounded 

□  Addresses  all  life 
cycle  stages 


Adapted  from  a  slide  by  Joe  Jarzombek  who,  in  turn,  credited  IEEE  CS  alternative 
proposal  for  15026  and  CMU  SEI  QUASAR  tutorial  by  Donald  Firesmith,  March  2007 
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Summary 


■  The  systems  engineering  challenge  for  systems  assurance  is  in 
integrating  a  heterogeneous  set  of  globally  engineered  and 
supplied  proprietary,  open-source,  and  other  software; 
hardware;  and  firmware;  as  well  as  legacy  systems;  to  create 
well-engineered  integrated,  interoperable,  and  extendable 
systems  whose  security,  safety,  and  other  risks  are  acceptable  - 
or  at  least  tolerable. 

■  Joint  industry  and  Government  efforts  are  ongoing  to  understand 
the  strengths  and  weaknesses  of  current  engineering  practices 
and  to  provide  appropriate  guidance 

■  National  and  international  standards  efforts  are  also  capturing 
and  codifying  minimum  acceptable  practice  regarding 
engineering  for  systems  assurance 

■  Systems  engineers  must  lead  the  way  in  sensitizing  their 
stakeholders  to  the  assurance  implications  of  engineering 
decisions  made  throughout  the  life  cycle  and  instill  practices  in 
their  own  engineering  organizations  that  facilitate  system 
assurance 
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For  More  Information 


Paul  R.  Croll 

Computer  Sciences  Corporation 

5166  Potomac  Drive 

King  George,  VA  22485-5824 

Phone:  +1  540.644.6224 
Fax:  +1  540.663.0276 

e-mail:  pcroll@csc.com 
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Overview 

•  Project  Overview 

•  Observations  on  the  requirements  and  the  current 
state  of  the  body  of  knowledge 
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Market  Segmentation 


Educating  the 
Acquisition  and  T&E 
Workforce  in  the  More 
Effective  Use  of  M&S: 

Market  Schema 


Career  Levels 


Advanced/Senior 

Intermediate/Journeyman 

Basic/Entry 


Acquisition  Career  Fields 

Program  Management 
Systems  Engineering 
Test  and  Evaluation 
Contracting 
Logistics 

Facilities  Engineering 
Auditing 

Science  &  Technology 

Information  Technology 

Business,  cost  estimating,  and  financial  mgmt 

Industrial  and/or  contract  property  management 

Manufacturing,  production  and  quality  assurance 

Purchasing 
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Experimentatiy.^^  ^(jucafi0n  for  Acquisition/T&E 
NDIA  Systems  Engineering  Conference 


Where  we  are  today 


•  Work  Completed 

-  DoD  M&S  Human  Capital  Strategy  -  COCOM  visits  conducted,  Survey  responses  being 
collected 

-  Consolidated  BOK  -  Drafted,  feedback  received  from  all  but  1  community 

-  Learning  Matrix  -  Draft  Complete 

•  Educational  catalog  delivered 

•  ESRs  developed 

•  Initial  gap  analysis  complete 

•  Academic  partners  identified  and  participating  (GMU,  JHU/APL,  ODU,  UAH,  UCF,  and  UCSD) 

•  Near  Term  Activities  -  Through  December  2007 

-  Complete  Spiral  One 

•  Incorporate  Stakeholder  feedback  to  Learning  Matrix  (10  October) 

•  Shaping  Meeting  (10  October) 

•  BoK/HCS  completion  (expected  completion  1  Dec) 

-  Spiral  Two:  Learning  Architecture/lnstructional  Framework  Development 

•  Develop  module/syllabi  framework  based  on  learning  matrix 

•  FY08  Funding  will  enable  development  of  education  and  evaluation  program 
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Overall  Approach: 
Information  Trade  Space 
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High  Level  ESR  Development 


•  Process: 

-  Initial  list  of  ESR’s  developed  by  stakeholders  and  NPS  inter-disciplinary  team. 

-  Stakeholders  involved  in  iterative  process  to  expand  and  refine  ESR’s  to  current  list. 

•  Results: 

-  17  Process  ESR’s  -Focused  on  the  process  of  choosing  when  to  use  which  models 
and  simulations. 

-  9  Acquisition  ESR’s  -Focused  on  applying  M&S  in  the  acquisition  lifecycle. 

-  5  Test  and  Evaluation  ESR’s  -Focused  on  the  role  and  use  of  M&S  in  test  and 
evaluation. 

-  5  Operational  ESR’s  -Focused  on  the  use  of  operational  and  logistic  M&S  to  support 
Acquisition/T&E  activities. 

-  14  Engineering  ESR’s  -Focused  on  the  use  of  engineering  models  to  support 
Acquisition/T&E  activities. 

Note:  High  Level  ESR’s  listed  in  backup  slides. 
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Workforce  Mapping 


•  Mapping  of  ESRs  to  workforce  needs  (Learning  Matrix) 

•  Performed  by  Academic  Partners,  including  GMU,  JHU/APL,  ODU,  UAH,  UCF,  and 
UCSD 

•  Three  pieces  provided  to  complete  mapping: 

Workforce  segmentation  definitions 

•  Career  Fields  -  Project  Managers,  Systems  Engineers,  and  T&E  workforce 

•  Career  Levels  -  Basic/entry,  intermediate/journeyman,  and  advanced/senior  career  levels 

•  Follows  DoD  5000.52M  descriptions 

Competence  Levels 

•  Four  competence  levels  defined  and  mapped  to  Bloom’s  taxonomy  -  General  Awareness, 
Understand,  Application,  and  Mastery 

Detailed  ESR’s  -  High  level  ESR’s  decomposed  into  “mappable”  level  of  granularity 

Note:  Career  field/level  descriptions  and  competence  level  descriptions  provided  in 
backup  slides. 
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JEiSiL  Workforce  Mapping  Example 

i  i  \J  I 

^ W~  Learning  Matrix  for  one  ESR  (of  50) 


P13:  Understand  the  trades  between  using  a  general  model  and  a  custom  model,  including  the  VV&A  implications. 

P13.1 

P13.2 

P13.3 

P13.4 

P13.5 

P13.6 

P13.7 

P13.8 

P13.9 

PM 

Basic 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

Intermediate 

Understand 

Application 

Application 

Application 

Application 

Application 

Application 

Mastery 

Mastery 

Advanced 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

SE 

Basic 

Understand 

Understand 

Understand 

Under: 

tttanrl 1 

1  1  InHorctanH  1 

1  1  InHorctanH  1 

1  1  InHorctanH  1 

1  1  InHaretanH 1 

1  1  InHorctanH 

PI  3.1  Define  general  model  and  custom  model 

PI  3.2  State  advantages  of  general  model 

PI  3.3  State  disadvantages  of  general  model 

PI  3.4  State  advantages  of  custom  model 

PI  3.5  State  disadvantages  of  custom  model 

PI  3.6  State  VVA  requirements  of  general  model 

PI  3.7  State  VVA  requirements  of  custom  model 

PI  3.8  Describe  situations  where  each  type  of  model  is 
more  appropriate 

PI  3.9  Given  historical  examples  of  each,  describe  and 
analyze  which  is  more  appropriate 

Intermediate 

Understand 

Application 

Application 

Applici 

Advanced 

Understand 

Application 

Application 

Applici 

T&E 

Basic 

Understand 

Understand 

Understand 

Under: 

Intermediate 

Understand 

Application 

Application 

Applici 

Advanced 

Understand 

Application 

Application 

Applici 
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Systen 

Current  work  (complete  Dec  07) 


Goal  -  Develop  Course/Module  “Syllabi” 

Syllabi  outline  desired  content  of  educational  elements  that  will  satisfy  the  needs 
identified  in  the  Learning  Matrix. 

Syllabi  combined  into  a  consolidated  and  cohesive  Learning  Architecture. 

Each  module  developed  to  highest  level  of  competency  required  for  the  subject  matter 
(not  always  mastery) 

Modules  constructed  so  that  slices  of  the  content  can  be  extracted  for  lower  required 
competency  levels 


General  Awareness  ( i.e.  W  hrs) 

i  i 

Understand  ( i.e.  X  hrs) 


Application  ( i.e.  Y  Hrs) 


„  ...  Mastery  ( i.e.  Z  hrs) 
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NDIA  Systems  Engineering  Conference 


Next  phase 


•  Courses  built  to  target  audience 

-  Desired  length  of  courses  and  competency  levels 
required  determine  subset  of  modu  es  combined  into 
course  structure 

-  Human  Capital  Strategy  survey  feedback  will  help  guide 
requirements. 

•  Courses  tested 
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Academic  Partners 


•  Air  Force  Institute  of 
Technology 

•  Defense  Acquisition 
University 

George  Mason  University 

Johns  Hopkins  University/ 
Applied  Physics  Lab 

Old  Dominion  University 


•  Stevens  Institute 

•  Texas  A&M 

University  of  Alabama, 
Huntsville 

University  of  California, 
San  Diego 

University  of  Central 
Florida 
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Process 


PI)  Understand  the  critical  decisions  in  the  acquisition  lifecycle,  the  analysis  plans  to  support 
them,  and  the  information  required. 

P2)  Understand  the  role  of  modeling  and  simulation  prior  to  the  concept  decision  to  identify  and 
quantify  capability  gaps  and  to  estimate  how  well  new  program  concepts  might  address 
those  gaps. 

P3)  Understand  the  costs,  benefits,  and  risks  of  using  physical  testing,  modeling  and  simulation, 
and  historical  data  to  provide  information  for  acquisition  decisions. 

P4)  Know  the  technical  aspects  of  the  domain  of  application. 

P5)  Know  the  taxonomy  and  hierarchies  of  models  and  simulations  and  be  able  to  select 
appropriately  for  a  given  situation.  Understand  the  types  of  architectures  and  role  of 
architectures  in  tying  together  and  communicating  requirements,  analysis,  modeling  and 
simulation,  design,  and  development  planning  to  all  stakeholders.  Understand  how  M&S 
is  deployed  in  different  environments  (Live,  Virtual,  and  Constructive).  Understand  the 
differences  between  standalone  and  confederated  M&S  applications  and  when  to  apply 
each  in  various  situations.  Be  familiar  with  the  simulation  interoperability  standards. 
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Process 


P6)  Establish  and  write  valid  modeling  and  simulation  requirements  using  a  process  that 
includes  modeling  and  simulation  needs  analysis,  generation  of  valid  modeling  and 
simulation  requirements,  functional  decomposition  and  conceptual  model  development, 
and  issuance  of  “built  to”  or  “buy  to”  performance  specifications.  Understand  how  models 
and  simulations  evolve  in  fidelity,  resolution,  and  scope  as  the  program  life  cycle 
progresses. 

P7)  Estimate  the  cost,  develop  a  schedule,  and  measure  the  performance  of  a  modeling  and 
simulation  plan.  Identify  the  areas  of  risk  and  develop  a  mitigation  strategy. 

P8)  Know  how  to  incorporate  modeling  and  simulation,  through  a  Simulation  Support  Plan,  into  a 
systems  engineering  plan  and  a  test  and  evaluation  master  plan. 

P9)  Know  and  require  the  best  practices  and  standards  in  modeling  and  simulation  as  developed 
in  key  case  studies. 

P10)  Know  the  models  and  simulations  used  in  a  given  domain,  their  inputs  and  outputs,  and 
their  strengths  and  weaknesses. 
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Process 


Pll)  Know  the  common  terminology  and  high  level  roles  and  responsibilities,  as  well  as  the 
underlying  philosophy,  principles,  and  methodologies  used  in  W&A  efforts,  especially 
those  applied  in  DoD. 

P12)  Be  able  to  correctly  match  the  level  of  detail  of  a  model  with  that  of  the  information  needed 
to  support  a  decision,  and  understand  the  connection  between  the  decision  to  be  made 
and  the  estimation  of  measures  from  the  model. 

P13)  Understand  the  trades  between  using  a  general  model  and  a  custom  model,  including  the 
W&A  implications. 

P14)  Design  a  sound  simulation  study  for  a  given  set  of  objectives. 

P15)  Apply  appropriate  statistical  techniques  to  the  analysis  of  simulation  output. 

P16)  Know  how  to  manage  and  reuse  existing  models,  data,  and  simulations  appropriately  and 
assure  that  new  products  developed  are  designed  and  prepared  for  reuse. 

P17)  Manage  the  data  strategy  for  an  M&S  effort  including  estimating  the  resources  necessary 
to  obtain  sufficient  data  to  populate  the  model. 
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Acquisition 


Al)  Understand  the  types,  role  and  value  of  formal  Modeling  and  Simulations,  and  their  various 
characterizations  for  application  to  systems  management,  particularly  with  regard  to 
design,  testing,  training,  production,  cost  estimation,  manning,  and  logistical  simulations. 

A2)  Understand  the  concepts  of  Simulation-Based  Acquisition  (SBA)  across  the  entire  program 
life  cycle,  in  order  to  reduce  the  time,  resources,  and  risks  associated  with  the  acquisition 
process. 

A3)  Be  able  to  discern  among  M&S  proposals,  relative  to  measurable  program  contributions,  and 
decide  on  the  appropriate  program  office  level  of  expenditure  on  M&S  tools  throughout 
the  program  life  cycle.  Distinguish  whether  custom  or  off-the-shelf  products  will  be  best 
suited  for  the  program’s  purpose. 

A4)  Understand  the  role  of  M&S  in  the  contract  proposal  process,  how  M&S  efforts  will  be 

defined  and  specified,  and  the  value  of  M&S  deliverables  under  an  acquisition  contract. 
Determine  their  need  for  continuous  improvement,  vis-a-vis  M&S  cost/benefit  trades 
throughout  the  program  life  cycle. 


23  Oct  2007 


M&S  Education  for  Acquisition/T&E 
NDIA  Systems  Engineering  Conference 


15 


Acquisition 


A5)  Know  where  to  find  organizational  M&S  resources  to  identify  the  number  and  types  of 
models  currently  in  use,  best  practices  from  case  studies,  where  they  originated,  how 
they  might  be  leveraged  in  support  of  an  acquisition  program. 

A6)  Be  aware  of  the  Modeling  and  Simulation  Resource  Repository  as  a  single  source  for 

information  about  and  access  to  DoD  models,  simulations,  data  sources,  algorithms,  and 
other  M&S  resources  in  order  to  facilitate  reuse  and  avoid  duplication. 

A7)  Understand  experimental  design,  level  of  model  detail,  and  M&S  application  as  a  pre-test 
prediction  tool.  Use  M&S  to  make  informed  engineering  tradeoff  analyses  through  the 
program’s  Decision  Risk  Analysis  process.  Understand  the  analysis  of  M&S 
outputs/measures. 

A8)  Understand  the  critical  interrelationships  and  balance  between  modeling  and  simulation  and 
more  traditional  forms  of  test  and  evaluation  (T&E)  -  particularly  operational  and  live-fire 
test  and  evaluation. 

A9)  Know  how  to  employ  M&S  to  explore  reliability  and  interoperability  issues. 
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Test  and  Evaluation 


Tl)  Quantify  the  risk  of  using  M&S  in  place  of  live  testing.  For  open  systems,  quantify  the  risk  of 
using  M&S  to  evaluate  a  single  system  component  in  place  of  testing  an  entire 
configuration. 

T2)  Integrate  M&S,  live  test,  prototype  data,  historical  data,  component  data,  and  scale  model 
data  into  a  coherent  testing  decision. 

T3)  Understand  the  different  types  of  testing  (i.e.  unit,  integration,  interoperability,  and 
operational)  and  identify  the  utility,  limitations  and  risks  for  use  of  M&S  in  each. 

T4)  Understand  the  potential  opportunities  for  employing  M&S  in  the  test  planning  and  execution 
process. 

T5)  Be  aware  of  existing  M&S  T&E  facilities  used  within  the  DoD. 
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Operational/Logistics 


01)  Understand  the  role  of  operational  and  logistical  models  in  the  acquisition  life  cycle  and 
when  they  are  used. 

02)  Know  the  properties  of  a  representative  suite  of  operational  models  across  the  services. 

Required  inputs,  Outputs,  Assumptions,  Implementation  requirements,  Costs,  Time 
required,  Adaptability  and  extensibility,  VVA  status 

03)  Know  the  properties  of  a  representative  suite  of  logistical  models  across  the  services. 

Required  inputs,  Outputs,  Assumptions,  Implementation  requirements,  Costs,  Time 
required,  Adaptability  and  extensibility,  VVA  status 

04)  Understand  abstractions  and  lower  levels  of  realism  in  operational  and  logistics  models. 

05)  Understand  and  be  able  to  model  the  components  of  logistics  systems,  including  Supply 
Chain,  Storage  systems,  Facilities,  Production,  Inventory  management,  Transportation  & 
distribution,  Replenishment  policies 
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Engineering/T  echnology 


El)  Structural  Mechanics,  Shock  and  Vibrations  -  Understand  basic  structural  mechanics 
including  stress-strain  relations,  buckling  and  fatigue,  shock  and  vibration,  and  finite 
element  methods  in  M&S. 

E2)  Fluid  Dynamics  and  Weapon  System  -  Understand  the  basics  of  computational  fluid 
dynamics  for  CFD  application  and  use  for  M&S.  Fluid  dynamics  of  subsonic  and 
supersonic  weapons,  warheads  and  their  effects. 

E3)  Dynamics  and  Control  -  Understand  the  basics  of  M&S  in  process  and  multi-physics 
(mechanical,  electrical  &  hydraulic)  based  dynamic  system  controls. 

E4)  Thermodynamics  and  Heat  Transfer  -  Understand  the  fundamentals  of  thermodynamics  and 
heat  transfer  with  applications  to  M&S  in  engineering  power  cycles,  propulsion  and 
auxiliary  system  cycle  analysis  and  design. 

E5)  Materials  and  Fabrication  -  Possess  a  basic  understanding  of  the  materials  technology 
associated  with  manufacturing,  welding  and  corrosion  control.  Have  an  introduction  to 
composite,  superconducting  materials,  and  fiber  optics  as  applied  to  M&S. 
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Engineering/T  echnology 


E6)  Acoustic  and  Electromagnetic  Systems  -  Have  a  general  awareness  of  the  fundamentals  of 
acoustic  and  electromagnetic  wave  propagation  and  application  to  DoD  systems. 

E7)  Military  Platform  Systems  Engineering  -  Appreciate  the  broad-based  design  oriented  M&S 
approach  for  complex  platforms  that  interact  with  air-land-sea-based  hardware  systems, 
command  and  control  systems  and  combat  systems. 

E8)  Computers  -  Recognize  basic  computer  system  architecture,  operating  systems,  networking 
and  introduction  to  engineering  software  and  their  applications.  Have  an  introduction  to 
structured  programming  languages  such  as  Fortran  and  C,  and  the  use  of  such  tools  for 
code  development.  Gain  exposure  to  finite  element/difference  codes,  with  application  to 
solve  engineering  problems  including  experience  with  selected  software  packages. 

E9)  Electrical  Engineering  -  Understand  basic  circuit  analysis  including  DC  and  AC  circuits.  Gain 
an  exposure  to  the  construction  and  operating  characteristics  of  rotating  machinery,  static 
converters,  power  distribution  systems  and  multi-phased  circuits. 
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Engineering/T  echnology 


E10)  C4ISR  -  Value  the  requirement  for  C4ISR  in  systems.  Understand  the  basic  components, 
methods  and  alternatives  for  transferring  information  from  one  point  to  another  both 
internal  and  external  to  the  system  being  considered.  Have  the  ability  to  analyze  all 
available  technologies  for  achieving  rapid/effective/jam-resistant  information  transfer. 

Ell)  Networks  -  Understand  the  principles  of  networks  applied  to  military  applications  including 
physical,  command  and  control,  and  social  networks  and  their  implications  for 
engineering  design  of  systems. 

E12)  Environment  -  Understand  the  fundamentals  of  terrestrial  science  (geology,  oceanography, 
meteorology,  and  near-earth  space  science)  to  describe  how  systems  interact  with  and  " 
are  influenced  by  their  environment. 

E13)  Human  Systems  Integration  -  Understand  the  principles  of  Human  Systems  Integration. 
Describe  the  applications  of  M&S  to  support  HSI  design  and  analysis. 

E14)  Aerodynamics  -  Understand  the  principles  of  aerodynamics  with  applications  to  M&S. 
Understand  the  cost,  schedule,  and  iterative  development  nature  of  simulation  testbeds 
used  for  flight  software  development  through  formal  qualification. 
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Conclusions 
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Program  Management  Career  Field 

(from  DoD  5000.52M) 


•  Positions  Held: 

-  All  of  functions  of  a  PMO  or  PEO 

-  Program  integrators  and  analysts,  program  managers,  PEOs,  and 
deputies 

-  Support  and  management  positions  throughout  the  workforce 

•  Responsibilities: 

-  Balance  the  factors  that  influence  cost,  schedule,  and 
performance 

-  Interpret  and  tailor  application  of  the  DoD  5000  Series  regulations 

-  Ensure  that  high-quality,  affordable,  supportable,  and  effective 
defense  systems  are  delivered  to  the  warfighter  as  quickly  as 
possible 
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PM  Career  Levels 


•  Basic/Entry 

-  Member  working  in  PM  support  role 

-  Example  jobs  include  R&D  coordinator,  test  officer  staff  officer, 
integrator,  analyst,  etc. 

•  Intermediate/Journeyman 

-  Managers  of  PEO/PMO  office  functions 

-  Deputy  PM  or  PM  for  small  programs,  PEO  staff  roles 

•  Advanced/Senior 

-  ACAT 1  or  2  PM,  PEO 
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SPRDE  -  Systems  Engineering 


Career  Field 


(from  DoD  5000.52M) 


•  Positions  Held: 

-  Scientists  and  engineers  supporting  science  and  technology  and 
acquisition  programs,  projects,  or  activities  to  accomplish  the 
responsibilities  below 

•  Responsibilities: 

-  Planning,  organizing,  monitoring,  managing,  overseeing  and/or 
performing  research  and  engineering  activities  relating  to  the 
design,  development,  fabrication,  installation,  modification, 
sustainment,  or  analysis  of  systems  or  systems  components 
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SE  Career  Levels 


•  Basic/Entry 

-  Design,  development,  fabrication,  installation,  modification, 
sustainment,  or  analysis  of  systems  or  systems  components 

-  Workforce  executing  these  tasks  across  a  broad  range  of 
application  domains 

•  Intermediate/Journeyman 

-  Managers  of  teams  working  on  application  level  functions 

-  Line  managers  at  warfare  centers,  project  leads  for  R&D  projects 

•  Advanced/Senior 

-  CHENG,  Warfare  center  execs,  project  systems  engineers 
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Test  and  Evaluation  Career  Field 

(from  DoD  5000.52M) 


•  Positions  Held: 

-  T&E  team  members;  T&E  leads  for  programs;  Service,  Agency, 
and  Facility  T&E  managers,  engineers,  scientists,  operations 
research  analysts,  system  analysts,  computer  scientists;  other 
technical  personnel  who  plan,  perform,  and  manage  T&E  tasks  in 
support  of  acquisition 

•  Responsibilities: 

-  Plan,  monitor,  manage,  and  conduct  T&E  of  prototype,  new, 
fielded,  or  modified  C4ISR  and  weapon  or  automated  information 
systems,  equipment  or  material 

-  Analyze,  assess,  and  evaluate  test  data  and  results  and  prepare 
assessments  of  system  performance  and  reports  of  T&E  findings 
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T&E  Career  Levels 


•  Basic/Entry 

-  Test  planning,  execution,  and  analysis  functions  performed  for 
DT  and  OT,  including  interoperability  and  other  certification 
testing  requirements 

-  Includes  testing  in  Live,  Virtual  and  Constructive  environments 

-  Workforce  executing  these  tasks  across  a  broad  range  of 
application  domains 

•  Intermediate/Journeyman 

-  Managers  of  testing  activities 

-  T&E  agency  team  leads,  Project  T&E  managers 

•  Advanced/Senior 

-  Testing  activity  executives,  PMs,  Milestone  decision  authorities 
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Competence  Levels 


Competence 

Level 

Bloom’s 

Taxonomy 

Definition 

Examples  and  Keywords 

General 

Awareness 

Knowledge 

Recall  or  recognize  data  or 
information. 

Examples:  Recite  a  policy.  Quote  prices  from  memory  to  a 
customer.  Knows  the  safety  rules. 

Keywords:  defines,  describes,  identifies,  knows,  labels,  lists, 
matches,  names,  outlines,  recalls,  recognizes,  reproduces, 
selects,  states. 

Understand 

Comprehension 

Understand  the  meaning, 
translation,  interpolation, 
and  interpretation  of 
instructions  and  problems. 
State  a  problem  in  one's 
own  words. 

Examples:  Rewrites  the  principles  of  test  writing.  Explain  in  one's 
own  words  the  steps  for  performing  a  complex  task.  Translates  an 
equation  into  a  computer  spreadsheet. 

Keywords:  comprehends,  converts,  defends,  distinguishes, 
estimates,  explains,  extends,  generalizes,  gives  Examples,  infers, 
interprets,  paraphrases,  predicts,  rewrites,  summarizes, 
translates. 

Application 

Application 

Use  a  concept  in  a  new 
situation  or  unprompted 
use  of  an  abstraction. 

Applies  what  was  learned 
in  the  classroom  into  novel 
situations  in  the  work  place. 
Put  theory  into  practice, 
use  knowledge  in  response 
to  real  circumstances 

Examples:  Use  a  manual  to  calculate  an  employee's  vacation 
time.  Apply  laws  of  statistics  to  evaluate  the  reliability  of  a  written 
test. 

Keywords:  applies,  changes,  computes,  constructs, 
demonstrates,  discovers,  manipulates,  modifies,  operates, 
predicts,  prepares,  produces,  relates,  shows,  solves,  uses. 

References: 
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Competence  Levels 


Competence 

Level 

Bloom’s 

Taxonomy 

Definition 

Examples  and  Keywords 

Mastery 

Analysis 

Separates  material  or 
concepts  into  component  parts 
so  that  its  organizational 
structure  may  be  understood. 
Distinguishes  between  facts 
and  inferences. 

Examples:  Troubleshoot  a  piece  of  equipment  by  using  logical 
deduction.  Recognize  logical  fallacies  in  reasoning.  Gathers 
information  from  a  department  and  selects  the  required  tasks  for 
training. 

Keywords:  analyzes,  breaks  down,  compares,  contrasts, 
diagrams,  deconstructs,  differentiates,  discriminates,  distinguishes, 
identifies,  illustrates,  infers,  outlines,  relates,  selects,  separates. 

Synthesis 

Builds/develops  new 
structures,  systems,  models, 
approaches,  or  patterns  from 
diverse  elements.  Put  parts 
together  to  form  a  whole,  with 
emphasis  on  creating  a  new 
meaning  or  structure. 

Examples:  Write  a  company  operations  or  process  manual. 

Design  a  machine  to  perform  a  specific  task.  Integrates  training 
from  several  sources  to  solve  a  problem.  Revises  and  process  to 
improve  the  outcome. 

Keywords:  categorizes,  combines,  compiles,  composes,  creates, 
devises,  designs,  explains,  generates,  modifies,  organizes,  plans, 
rearranges,  reconstructs,  relates,  reorganizes,  revises,  rewrites, 
summarizes,  tells,  writes. 

Roforoncos: - 

Evaluation 

Make  judgments  about  the 
value  of  ideas  or  materials. 
Assess  effectiveness  of  whole 
concepts  in  relation  to  values, 
outputs,  efficacy,  viability; 
critical  thinking,  strategic 
comparison  and  review. 

Examples:  Select  the  most  effective  solution.  Hire  the  most 
qualified  candidate.  Explain  and  justify  a  new  budget. 

Keywords:  appraises,  compares,  concludes,  contrasts,  criticizes, 
critiques,  defends,  describes,  discriminates,  evaluates,  explains, 
interprets,  justifies,  relates,  summarizes,  supports. 
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http://www.businesspalls.eom/bloomstaxonomvoflearninqdomains:htm 
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OSD  Software  Engineering 
and  System  Assurance 

•  Support  Acquisition  Success 

•  Improve  State-of-the-Practice 
of  Engineering 

•  Leadership,  Outreach  and 
Advocacy 

•  Foster  Resources  to  Meet 
DoD  Needs 
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Recap  of  Activities  Since  Last  NDi A  Conference 


•  Documented  software  issues,  needs 

-  Software  I  ndustrial  Base  Study  completed 

-  NDI  A  Top  Software  I  ssues  Workshop  Report 

-  Defense  Software  Strategy  Summit  Report 

•  Created  partnerships 

-  Established  network  of  DoD  software  POCs 

-  Chartered  the  NDI  A  Software  Committee  and  Expert  Panel 

-  Bi-weekly  software  collaboration  exchanges  with  Government,  Academia, 
and  I  ndustry 

-  Restructured  the  US- UK- AUS  Trilateral  Working  Group 

•  Performed  gap  analysis 

-  Identified  ongoing  software  initiatives;  mapped  them  to  issue  areas 

-  Two  outcomes: 

1.  Identified  initiatives  that  deserve  cross-DoD  attention 

2.  I  dentified  gaps  where  attention  is  needed 


Common  Goal:  Provide  visibility  to  key  initiatives; 

Focus  attention  on  gaps 


SW  Issue/  CAP  Workshop  Findings 

March  2007 
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Identified  Software  Gap  Areas 


•  Estimation 

•  Risk  I  dentification 

•  Sustainment 

•  I  nteroperability 

•  Test 

•  Requirements 

•  Quality  Attributes 

•  Qualifications  for  Software  Support  (SETA) 

•  Human  Capital  Strategy 


Needed  step: 

Develop  plans  to  define,  and  begin  to  address,  these  gaps 
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Software  In  Acquisition  Workshop 
October 16-17,  2007 


•  Purpose:  Off-year  workshop  (Summit  held  every  2yrs) 

-  Share  progress  on  initiatives  against  known  issues 

-  Collaborate  on  gaps 

•  Format: 

-  Leadership  updates 

-  Presentations  from  the  community  to  share  progress, 
experiences 

-  Workshops  to  develop  action  plans  for  resolution  of  key  issues 

•  Requirements,  Risk/ Cost  Estimation,  SW  Quality  Attributes 

•  Audience: 

-  DoD  programs,  practitioners,  industry,  FFRDC,  academia 

•  Community  forum  focused  on  software  in  acquisition 
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Software 

Requirements 

Workshop 
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Requirements  issues 


•  Many  problems  are  the  same  as  those  we  have  been  trying  to  fix  for 
30  years 

•  We  know  what  to  do  however  we  do  not  incentivize  it,  pay  for  it  and 
train  for  it  (e.g.  application  of  IEEE  requirements  attributes) 

•  Lack  of  early  and  continuous  involvement  of  all  relevant  SMEs 

•  We  are  making  changes  to  software  whose  existing  behavior  we  do 
not  understand 

•  Do  not  have  adequate  tools,  methods  and  processes  for 
requirements  definition 

•  How  are  articulated  specifications  adequate  to  development  derived 
from  capabilities?  -  especially  in  a  continuous  evolution  environment 

•  Requirements  are  added,  changed  or  deleted  without  sufficient 
engineering 

•  Component  cannot  operate  alone  -  no  one  owns  the  external 
relationships 
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Requirements  Workshop 
Recommendations 


1.  Define  an  effective  “software  portfolio"  management 
framework 

-  Protect  the  continuity  of  systems/ software  and  requirement 
engineering  throughout  the  software  life  cycle 

2.  I  implement  the  techniques  we  know  will  work  and 
identify  any  shortcomings 

-  Training 

-  I  ncentives 

-  Re-examine  I EEE  tenets  for  good  requirements 

3.  Find  ways  to  leverage  the  malleability  of  software 

-  We  need  new  methods  to  deal  with  "on  the  fly"  and/or  external 
requirements:  Software  has  the  ability  to  adapt  faster  than  other 
elements 

-  Build  and  integrate  effective  modeling  of  existing  systems  and 
addition  of  new  requirements 

-  Identify  resources  and  methods  to  facilitate  planning  for 
extended  use 

-  Find  ways  to  manage  the  malleability  to  minimize  risk 
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Requirements  Workshop 
Recommendations 


4.  Change  our  view/ perspective  of  "sustainment"  to 
"continuous  evolution" 

-  Codify  the  processes  for  “reverse  engineering"  candidates  to 
extract  for  reuse  -  system  components  from  government  or 
industry 

-  Look  at  organizational  as  well  as  methods  and  skills  to  perform 
continuous  evolution 

5.  Establish  a  research  program 

-  Identify  the  characteristics  of  requirements  engineering  in  type 
1 1 1  systems  and  how  it  is  distinguished  from  type  I 

-  Start  to  identify  good  practices 
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Software  Risk/  Cost 
Estimation 
Workshop 


SYSTEMS  AND  SOFTWARE  ENGINEERING  CENTER  OF  EXCELLENCE,  DUSD(A&T) 


Panel  #2:  Software  Estimation  /  Risk  - 

Top  Level  Summary 

•  Panel  Theme/  Focus: 

-  Policy,  guidance,  and  training  for  improving  software  acquisition  and 
program  execution  through  synergistic  integration  of  risk  management 
and  estimation  approaches 

•  2006  DoD  Software  Summit  Findings: 

-  Must  understand  that  software  is  a  primary  performance,  schedule,  and 
cost  driver 

-  Pressure  to  rapidly  procure  new  capabilities  can  inhibit  balance  of  life 
cycle  cost,  schedule,  and  performance  expectations  to  achieve 
executable  programs 

-  Software  risks  and  life  cycle  costs  are  not  consistently  accommodated  in 
planning 

-  Realistic  schedule  and  effort  or  cost  estimates  are  often  rejected  or 
constrained 

-  Reuse,  open  source,  and  government  off-the-shelf  software  estimating 
methods  are  inadequate 
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Software  Estimation/ Risk 
Recommendations  (1  of  2) 


•  Investigate  principles  for  organizing  WBS-related  artifacts  that  address 
Software  Engineering  sufficiently 

-  Concept  definition  6  mo,  Proposed  Language  12  mo,  Revised  WBS  for 
Mil-Hdbk-881  and  related  documents 

•  Investigate  strategies  and  approaches  for  developing  and  evolving  an 
integrated  software  data  repository  and  related  tools  to  address  a 
broad  set  of  stakeholders  (government,  industry,  academia) 

-  Concept  definition  and  definition  of  measures  6  mo,  data  collection  from 
sample  programs  12  mo,  Concept  of  Operations/Business  Plan  for  wide- 
scale  rollout  18  mo. 

•  Conduct  Root  Cause  analysis  studies  to  understand  the  problems  in 
software  estimation  and  the  use  of  estimates  in  the  acquisition 
process 

-  Data  gathering  of  lessons  learned  and  studies  6  mo,  Draft  analysis  12  mo, 
Prioritization  of  high  leverage  areas  for  improvement  18  mo 
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Software  Risk/  Estimation 
Recommendations  (2  of  2) 


•  Develop  and  implement  an  incremental  acquisition  approach  (as 
well  as  the  overall  acquisition  framework)  that  accommodates  the 
uncertainty  associated  with  early  software  estimates  and  allows  for 
adjustment  and  refinement  over  time 

-  Data  gathering  6  mo.,  analysis  of  data  12  mo.,  proposed  changes  to 
DoD  policy  and  guidance  documents  18  mo. 

•  Establish  policy,  related  guidance,  and  recommended 
implementation  approaches  for  software  data  collection  and  analysis 
across  all  DoD  acquisition  programs 

-  Concept  definition  6  mo,  initial  analysis  of  data  12  mo,  proposed 
changes  to  DoD  policy  and  guidance  documents  18  mo 
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Software  Quality 
Attributes 
Workshop 
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Software  Quality  Attribute 
Priority  Recommendations  (  1  of  2) 


1.  Product:  Recommended  guidance  on  engineering  issues  such  as: 
quantitatively  identifying,  predicting,  evaluating,  verifying,  and  validating 
Quality  Attributes 

Actions  to  include: 

1.1.  Address  tie-in  to  KPPs  and  TPMs 

1.2.  Identify  methods  for  quantitative  assessment  of  individual  and  integrated 
Quality  Attributes 

1.3.  Define  the  specific  pieces  of  evidence  required  to  pass  acquisition  milestones 

1.4.  I  dentify  methods  for  predicting  quality  attribute  outcomes  for  the  delivered 
system,  throughout  the  life  cycle 

2.  Product:  Recommendations  for  improving  OSD/  Service- level  acquisition 
policy  regarding  Quality  Attributes 

Actions  to  include: 

2.1.  I  dentify  benefits  of  addressing  software  quality  attributes  as  part  of  an 
acquisition  risk  reduction  strategy 

2.2.  Address  gaps  in  SEP,  TEMP,  J  Cl  DS,  DAG 

2.3.  Develop  model  RFP  language 

2.4.  Define  expectations  for  Quality  Attribute  review  during  Acquisition  Milestone 
Reviews  (e.g.  PDR) 
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Software  Quality  Attribute 
Priority  Recommendations  (2  of  2) 


3.  Product:  Taxonomy  of  software  quality  attributes  and  how  they 
are  related 

Actions  to  include: 

3.1.  Collect  and  organize  definitions  of  Quality  Attributes 

3.2.  Enumerate  relationships  to  systems  quality  expectations 

3.3.  Survey  existing  information  on  selection  and  prioritization  of  software 
quality  attributes  for  different  classes  of  programs 

4.  Product:  Program  Manager  guidance  on  I  ntroduction  to  Software 
Architecturafl Evaluation  of  Quality  Attributes 

Actions  to  include: 

4.1.  Evaluate  existing  guidance  documents 

4.2.  Synthesize  results  into  recommended  guidance 

5.  Product:  Collaboration  site  for  collecting  data,  sharing  work 
products,  facilitating  on-going  discussion 

Actions  to  include: 

5. 1.  I  dentify  host/ collaboration  tool 

5.2.  Define  site  framework/ rules 
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2nd  Tier  Work  Products 


1.  Product:  Catalog  of  architectural  approaches  with  respect  to  their  Quality 
Attribute  profiles 

Actions  to  include: 

1.1.  Develop  catalog  format  and  approach 

2.  Product:  Process  for  selecting  the  subset  of  Quality  Attributes  for  specific 
systems  of  interest 

Actions  to  include: 

2. 1.  Develop  strategy  for  attribute  trade-offs 

2.2.  Identify  risk  implications 

2.3.  Develop  a  checklist  of  questions  to  identify  attributes  important  to  the 
stakeholder  s) 

3.  Product:  Recommendations  for  basic  research  on  quality  attributes 

Actions  to  include: 

3.1.  Address  inadequacies  in  state  of  the  arty  state  of  the  practice 

4.  Product:  Guidance  on  how  to  engineer  quality  attributes  into  systems 

Actions  to  include: 

4.1.  Define  engineering  processes  to  achieve  specific  quality  attribute  levels 

4.2.  Report  on  current  research  and  practice 
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3rd  Tier  Work  Products  (1  of  2) 


1.  Product:  Guidance  on  addressing  Quality  Attributes  in  the  Systems  of  Systems 
Context 

Actions  to  include: 

1.1.  Perform  mission  thread  analysis 

1.1.1.  Use  workshop  outline  (funded) 

1.2.  Define  systems  of  Systems  software  architecture  evaluation  approach 

2.  Product:  Examination  of  Root  Cause  Analysis  Workshop  data  with  respect  to 
Quality  Attributes  implications 

Actions  to  include: 

2.1.  Examine  root  cause  analysis  workshop  data  to  determine  quality  attribute  implication 

3.  Product:  Examination  of  what  DAU  teaches  regarding  Quality  Attributes  and 
recommendations  for  improvement  (tied  to  policy  and  guidance  in  #2) 

Actions  to  include: 

3.1.  Review  course  material  used  for  PMs  and  Systems  Engineers  about  quality  attributes 

3. 1.1.  Provide  recommendations  for  additions  to  course  materials 

4.  Product:  White  paper  on  how  to  reason  about  Quality  Attributes  in  architecture 
model  standards  (e.g.  DODAF) 

Actions  to  include: 

4.1.  Produce  white  paper  on  how  to  reason  about  Quality  Attributes  in  architecture  model 
standards  (e.g.  DODAF) 
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3rd  Tier  Work  Products  (2  of  2) 


5.  Product:  Guidance  on  addressing  quality  attributes  of  COTS/  NDI 

Actions  to  include: 

5.1.  Develop  guidance  on  addressing  quality  attributes  of  COTS/NDI 

6.  Product:  White  paper  on  quality  attributes  implications  of  agile 
methods  for  large  scale  defense  systems 

Actions  to  include: 

6.1.  Develop  White  paper  on  quality  attributes  implications  of  agile  methods  for 
large  scale  defense  systems 

7.  Product:  Guidance/  lessons  learned  from  commercial  practice 

Actions  to  include: 

7.1. Collect  and  provide  guidance/ lessons  learned  from  commercial  practice 
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OUSD(A  T&D/SSA 
FOCUSED  INITIATI VES 
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System  Assurance  Guidebook 


Project  Description 


Issue:  Systems  are  vulnerable  to  malicious  tampering 


•  Project  Description: 

-  Provide  practical  guidance  on  augmenting  systems  engineering  practice  for 
system  assurance 

-  Synthesize  existing  knowledge  from  organizations,  standards  and  best 
practices 

-  Recap  concepts  from  standards 

•  Opportunity  for: 

-  Practitioners,  academe  who  implement  systems  engineering,  assurance, 
safety,  security,  program  protection,  etc.  into  processes  and  programs 

•  The  project  addresses 

-  I  ntegration  of  assurance  guidance  and  practices  into  systems  engineering 

•  Product: 

-  Guidebook  on  Engineering  for  System  Assurance 

•  Outcome  Goal: 

-  I  ntent  is  to  yield  assured  program  /  system  with  demonstrable  evidence  of 


assurance 
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System  of  Systems 
Project  Description 

Issue:  No  common  definition,  or  guidance  for  SoS 

•  Project  Description: 

-  Effort  led  by  the  Office  of  the  Secretary  of  Defense 

-  Collaborative  Approach  with  DoD,  I  ndustry.  Academia 

•  Purpose 

-  6  month  effort  addressing  areas  of  agreement  across  the  community 

-  Focus  on  technical  aspects  of  SE  applicable  across  SoS  management  constructs 

-  Vehicle  to  capture  and  debate  current  SoS  experience 

•  Audience 

-  Program  Managers  and  Lead/Chief  Engineers 

•  The  project  addresses 

-  Considerations  for  engineering  above  a  system  level 

•  Product: 

-  SoS  Engineering  Guide,  vl.O,  Fall  2007 

•  Outcome  Goal: 

-  Program  managers/ chief  engineers  have  requisite  knowledge  to  manage  SoS 
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SW  Engineering  Graduate  Curriculum 

Project  Description 


Issue:  There  is  no  commonly  accepted  structure  or 
content  for  graduate  software  engineering  education 


•  Project  Description: 

-  Develop  a  core  curriculum  and  core  competencies  for  software  engineering 

•  Opportunity: 

-  I  ndustrial  and  government  workforce  customers  of  SWE  graduate  education 

-  Academics  who  provide  SWE  and  SE  graduate  education 

-  Professional  societies  with  a  vested  interest  in  SWE  and  SE  graduate  education 

•  The  project  addresses 

-  I  nconsistencies  in  software  graduate  degrees 

-  Poor  definition  of  labor  categories  and  software  expertise 

-  The  divide  between  systems  and  software  engineers  in  industry,  government,  and 
academia 

-  The  project  will  integrate  SE  principles  and  practices  into  a  SWE  curriculum. 

•  Product: 

-  An  approved  curric  that  can  be  adopted  by  the  community  (industry,  academia, 
associations) 

•  Outcome  Goal (s): 

-  Software  engineers  have  a  more  consistent  training  base 
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DoD  Acquisition  Workforce  Software  Education 

Project  Description 


Issue:  DAWIA  Curriculum  does  not  address  software  acquisition  issues 

•  Project  Description: 

-  Compare  identified  software  competencies  with  current  curriculum 

-  Update  DAU  software  acquisition  management  courseware  and  other 
career  field  training  to  meet  competency  needs 

-  Develop  continuous  learning  modules  as  part  of  the  DAU  Core  Plus 
construct 

-  I  nitial  focus  on  PM  and  SPRDE  career  fields 

•  Opportunity  for: 

-  Software  and  career  field  process  owners  and  practitioners 

•  The  project  addresses 

-  Methodology  for  determining  software  competencies 

-  Methodology  for  developing  tailored  solutions  for  each  career  field 

•  Product: 

-  Updated  DAWIA  software  competencies  reflecting  latest  policies  and 
guidance 

•  Outcome  Goal: 

-  Acquisition  professionals  have  requisite  software  knowledge 
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SE/SW  Process  Integration 
Project  Description 


Issue:  SE  and  SW  have  not  been  well  integrated  on  projects 


•  Project  Description: 

-  Study  SE  and  SW  processes,  capture  ongoing  harmonization  efforts 

-  Assess  current  guidance 

-  Identify  opportunities  for  better  integration 

•  Opportunity  for: 

-  SE  and  SW  process  owners  or  practitioners 

-  Academe  who  teach/ study  SE  and  SW 

•  The  project  addresses 

-  I  ntegration  of  SW  with  requirements,  risk  management  and  other  SE 
technical  and  management  processes 

•  Product: 

-  Report  and  recommendations  for  SW  policy,  guidance,  and  tools  to 
better  integrate  with  SE  and  Acquisition 

•  Outcome  Goal: 

-  Software  is  a  major  factor  in  engineering  design  and  acquisition 
management  decisions 
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CMMI -Acquisition 
Project  Description 

Issue:  Acquirers  lack  an  appraisable  model  for  acquisition  PI 


•  Project  Description: 

-  Using  GM  CMMI  for  Outsourcing;  pilot  and  generate  CMMI-ACQ 

-  I  nvolve  broad  set  of  acquisition  stakeholders  to  ensure  wide  application 

•  Opportunity  for: 

-  Process  improvement  stakeholders 

-  Acquiring  organizations 

•  The  project  addresses 

-  Identification  of  key  acquirer  activities  and  products 

-  Amplification  of  CMMI  core  practices  to  capture  acquirer  considerations 

•  Product: 

-  CMMI  model  for  Acquisition  (built  on  CMMI  foundation  for  consistency 
with  CMMI -DEV) 

•  Outcome  Goal: 

-  Acquisition  organizations  implement  best  practices,  and  institute 
organizational  process  improvement 
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Additional  areas  for 
collaboration. . .  and  attention 

•  Additional  projects  we  are  looking  into: 

-  Software  earned  value  guidance 

-  Software  metrics 

-  Software  knowledge  portal 

•  Some  key  gaps  remaining: 

-  Software  sustainment 

-  Software  test 
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Our  Challenge 


•  Given  the  shortage  of  software  resources  and  critical  software 
reliance 

-  We  cannot  afford  to  be  stovepiped 

-  We  must  integrate  across  cross-functional  perspectives  to  improve  our 
software  capability 

•  We  must  focus  on  long  standing  software  issues 

-  Leverage  ongoing  activities  to  make  a  difference 

-  I  nvest  in  collaborative  efforts  where  there  are  gaps 

•  Now. . . 

-  Work  together  to  address  software  issues 

-  J  oin  the  DoD  SI  A  Collaborators  -  participate  in  bi-weekly  collaboration 
telecons 

-  Contribute  to  workshop  action  items,  and/or  ongoing  initiatives 

-  Contact  us  at  ATL-  SSA(a)osd .  mi  I 

Become  a  DoD  Center  of  Excellence 
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Backup  Slides 
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Elements  of  a  DoD  Strategy  for  Software 


•  Support  Acquisition  Success 

-  Ensure  effective  and  efficient  software  solutions  across  the  acquisition 
spectrum  of  systems,  SoS  and  capability  portfolios 

•  I  mprove  the  State-of- the- Practice  of  Software  Engineering 

-  Advocate  and  lead  software  initiatives  to  improve  the  state-of-the- 
practices  through  transition  of  tools,  techniques,  etc. 

•  Leadership,  Outreach  and  Advocacy 

-  I  implement  at  Department  and  National  levels,  a  strategic  plan  for 
meeting  Defense  software  requirements 

•  Foster  Software  Resources  to  meet  DoD  needs 

-  Enable  the  US  and  global  capability  to  meet  Department  software 
needs,  in  an  assured  and  responsive  manner 


Promote  World-Class  Leadership  for  Defense  Software  Engineering 
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Next  Steps 


Near  Term: 

•  Determine  metrics  for  each  of  the  6  Focus  Areas 

-  Based  upon  source  reports  (ie.  SW  Summit,  Top  Issues,  PSRs,  Historical  SW  Studies) 

•  Coordinate  ongoing  initiatives  (via  Working  Group  Participation,  Defense 
Software  in  Acquisition  Collaborators) 

-  Support  and/or  leverage  initiatives  where  appropriate 

-  Provide  visibility  across  the  Department 

•  Determine  action  plans  for  each  gap  considering: 

-  Priority 

-  Near  T errrY  Long  T erm  i  impacts 

-  NDIA  SW  Committee,  others,  interest  in  accepting  gap(s) 

•  Engage  other  communities  and  participants 

-  IT,  Business,  Research 

Over  Time: 

•  Reassess  ongoing  initiatives  against  focus  area  metrics 

-  Determine  new  gaps,  or  additional  effort  required  to  address  core  issues 

•  Reassess  focus  area  metrics  against  systemic  software  issues 

-  From  future  SW  Summits,  Systemic  Analysis,  etc. . . 


SYSTEMS  AND  SOFTWARE  ENGINEERING  CENTER  OF  EXCELLENCE,  DUSD(A&T) 


Top  Software  Issues * 


1.  The  impact  of  requirements  upon  software  is  not  consistently  quantified  and 
managed  in  development  or  sustainment. 

2.  Fundamental  system  engineering  decisions  are  made  without  full  participation 
of  software  engineering. 

3.  Software  life-cycle  planning  and  management  by  acquirers  and  suppliers  is 
ineffective. 

4.  The  quantity  and  quality  of  software  engineering  expertise  is  insufficient  to 
meet  the  demands  of  government  and  the  defense  industry. 

5.  Traditional  software  verification  techniques  are  costly  and  ineffective  for 
dealing  with  the  scale  and  complexity  of  modern  systems. 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure  execution  of 
complex  software  in  distributed  environments. 

7.  I  nadequate  attention  is  given  to  total  lifecycle  issues  for  COTS/NDI  impacts 
on  lifecycle  cost  and  risk. 


*NDIA  Top  Software  Issues  Workshop 
August  2006 
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DoD  Software 


What  We're  Seeing * 


•  Software  systemic  issues  are  significant  contributors  to  poor  program 
execution 

-  Software  requirements  not  well  defined,  traceable,  testable 

-  I  mmatu re  architectures,  COTS  integration,  interoperability,  obsolescence 
(electronics/hardware  refresh) 

-  Software  development  processes  not  institutionalized,  planning  documents 
missing  or  incomplete,  reuse  strategies  inconsistent 

-  Software  test/evaluation  lacking  rigor  and  breadth 

-  Schedule  realism  (compressed,  overlapping) 

-  Lessons  learned  not  incorporated  into  successive  builds 

-  Software  risks/ metrics  not  well  defined,  managed 


*  Based  on  —65  program  reviews  to  date 
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The  Integrated  Software  and  Systems 
Engineering  Curriculum  Project: 

Creating  a  Reference  Curriculum  for 
Graduate  Software  Engineering  Education 


Kristen  Baldwin  and  Art  Pyster 


October  23,  2007 


STEVENS 


Institute  of  lechnolo^y 


Office  of  the  Under  Secretary  of  Defense 
Acquisition,  Technology  and  Logistics 
Systems  and  Software  Engineering 


Stevens  Institute  of  Technology 
School  of  Engineering 
Applied  Systems  Thinking  Institute 


Background 


■  Software  drives  the  performance  of  virtually  all  major 
systems. 

■  Being  able  to  produce  software  that  can  be  trusted  as 
reliable,  secure,  safe,  correct,  and  available  while 
being  delivered  on-time  and  within  budget  is  a  major 
challenge  for  both  the  government  and  industry. 

■  Many  steps  must  be  taken  to  meet  that  challenge  - 
including  ensuring  our  workforce  is  well  educated  in 
software  engineering  (SWE)  principles  and  practices. 

■  Yet  today,  there  is  no  commonly  accepted  modern 
structure  or  content  for  graduate  software  engineering 
education.  Last  effort  was  in  early  1990s  by  the  SEI. 
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/SSEc-  The  Way  Forward 


The  Integrated  Software  and  Systems 
Engineering  Curriculum  Project  (iSSEc) 
is  creating  a  reference  curriculum 
leading  to  a  Masters  degree  in  software 

engineering 


iSSEc  -  The  Way  Forward 


■  iSSEc  is  sponsored  by  DOD  and  led  by  Stevens,  involving  4 
sets  of  stakeholders: 

■  The  industrial  and  government  workforce  who  are  the  customers  of 
SWE  graduate  education 

■  Academics  who  provide  SWE  and  SE  graduate  education 

■  Professional  societies  with  a  vested  interest  in  SWE  and  SE 
graduate  education 

■  Government  organizations  who  fund  improvements  in  SWE 
graduate  education 

■  iSSEc  recognizes  that  the  divide  between  systems  and  software 
engineers  in  industry,  government,  and  academia  works  against 
successfully  delivering  modern  systems  in  which  software  is 
almost  always  central. 

■  iSSEc  will  integrate  SE  principles  and  practices  into  the  SWE 
curriculum.  The  bright  line  that  now  separates  SE  and  SWE  in 
academia  must  be  eliminated! 
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The  Approach 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  a  strawman  model  curriculum,  suitable  for  broad 
use,  with  a  small  representative  team  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 

etc.  (continuous) 

4.  Gradually  obtain  endorsement  from  ACM,  IEEE, 
INCOSE,  NDIA,  and  other  professional  organizations 

(continuous) 

5.  Create  full  model  curriculum,  suitable  for  global  use,  with 

a  large  representative  team  (September  2008  and  September 
2009) 

6.  Seek  early  adopters  (continuous) 
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Status  -  Understand  Current  State 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  a  strawman  model  curriculum,  suitable  for  broad 
use,  with  a  small  representative  team  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 

etc.  (continuous) 

4.  Gradually  obtain  endorsement  from  ACM,  IEEE, 
INCOSE,  NDIA,  and  other  professional  organizations 

(continuous) 

5.  Create  full  model  curriculum,  suitable  for  global  use,  with 

a  large  representative  team  (September  2008  and  September 
2009) 

6.  Seek  early  adopters  (continuous) 
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Understanding  the  Current  State 


■  Select  diverse  set  of  universities  with  Masters  programs 
in  SWE  -  vary  in  size,  geography,  maturity,  resources, 
target  market,  . . . 

■  Use  Software  Engineering  Body  of  Knowledge 
(SWEBOK)  as  primary  framework  for  SWE  competencies 

■  Collect  data  from  school  websites 

Degree,  faculty  size,  student  population,  target  market,  . . . 
Degree  structure,  individual  course  descriptions 
Map  between  courses  and  SWEBOK 

■  Validate  data  with  professor 

■  Analyze  for  commonalities  and  uniqueness 


Schools  Completed  or  I  n  Process 


1 .  Air  Force  Institute  of  Technology 

2.  Brandeis  University 

3.  California  State  University  - 
Fullerton 

4.  California  State  University- 
Sacramento 

5.  Carnegie  Mellon  University 

6.  Carnegie  Mellon  University  West 

7.  Carrol  College 

8.  DePaul  University 

9.  Dublin  City  University  (Ireland) 

10.  Embry-Riddle  Aeronautical 
University 

11.  Florida  A&M 

12.  George  Mason  University 

13.  James  Madison  University 

14.  Kingston  University  (UK) 

15.  Mercer  University 


16.  Monmouth  University 

17.  Naval  Postgraduate  School 

18.  Rochester  Institute  of  Technology 

19.  Seattle  University 

20.  Southern  Methodist  University 

21 .  Stevens  Institute  of  Technology 

22.  Texas  Tech 

23.  University  of  Alabama-Huntsville 

24.  University  of  Colorado  -  Colorado 
Springs 

25.  University  of  Michigan  -  Dearborn 

26.  University  of  Quebec  (Canada) 

27.  University  of  Scranton 

28.  University  of  Southern  California 

29.  University  of  Sunderland  (UK) 

30.  University  of  York  (UK) 

Some  changes  still  likely 
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SWEBOK's  10  Knowledge  Areas 


REQ 

Software  Requirements 

DES 

Software  Design 

CST 

Software  Construction 

TST 

Software  Testing 

MNT 

Software  Maintenance 

CNF 

Software  Configuration  Management 

MGT 

Software  Engineering  Management 

PRC 

Software  Engineering  Process 

TLS 

Software  Engineering  Tools  and  Methods 

QLY 

Software  Quality 

Early  Observations  from  11  Schools 


1 .  SWE  is  largely  viewed  as  a  specialization  of  Computer 
Science  -  much  as  systems  engineering  was  often 
viewed  as  specialization  of  industrial  engineering  or 
operations  research  years  ago 

2.  Faculty  size  is  small  -  few  dedicated  SWE  professors, 
making  programs  relatively  fragile 

3.  Student  enrollments  are  generally  small  compared  to  CS 
and  to  other  engineering  disciplines 

4.  Many  programs  specialize  to  specific  markets  such  as 
defense  systems  or  safety  critical  systems 

5.  The  target  student  population  varies  widely  -  anyone 
with  Bachelors  and  B  average  to  someone  with  CS 
degree  and  2+  years  of  experience 
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More  Early  Observations 


6.  Program  outcomes  vary  widely  -  software  developer  to 
researcher  to  software  manager 

7.  Wide  variation  in  depth  and  breadth  of  SWEBOK 
coverage  in  required  and  semi-required  courses 

8.  SWEBOK  alone  does  not  represent  the  breadth  of  many 
program’s  required  courses 

9.  Some  significant  topics  are  rarely  mentioned  -  agility, 
Software  Engineering  Economics,  Systems  Engineering 

10.  Some  topics  are  ubiquitous  -  formal  methods  and 
architecture 

1 1 .  “Object-Oriented”  is  the  standard  development  paradigm  - 
creating  a  “clash”  with  many  systems  engineering 
programs  that  emphasize  structure  methods 


11 


Sample  Program  Specialty 


Air  Force  1  nstitute  of 

Technology 

Defense  Systems 

Embry- Riddle  Aeronautical 
University 

Embedded  Real-time  Software 

Naval  Postgraduate  School 

Acquisition  of  Defense  Systems 

Seattle  University 

Project  Experience 

Stevens  1  nstitute  of  Technology 

Quantitative  Software 

Engineering 

University  of  Southern 

California 

Quantitative;  Software 

Economics 

University  of  York  (UK) 

Safety  Critical  Systems 
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Sample  Program  Focus 


Air  Force  1  nstitute  of  Technology 

Develop  professionals  to  develop  and  manage  increasingly 
complex  software 

Embry- Riddle  Aeronautical  University 

How  to  engineer  high-performance  sofwtare  embedded  in 
aircraft,  space  and  medical  systems 

George  Mason  University 

Developing  and  modifying  large,  complex  software 
systems.  Emphasis  both  technical  and  management  aspects 

Monmouth  University 

Effective  member  of  software  development  team 

Naval  Postgraduate  School 

Enable  acquisition  professionals  to  procure  highly 
dependable,  trustworthy  software- intensive  systems 

Seattle  University 

Understand  and  apply  advanced  software  engineering 
principles  vital  to  industry 

Stevens  1  nstitute  of  Technology 

Realizing  software  products  on  time,  within  budget  and 
with  known  quality 

University  of  Alabama  -  Huntsville 

Provide  fundamentals  of  software  development  for 
members  of  software  development  teams 

University  of  Southern  California 

Prepare  students  for  an  industrial  leadership  career  in 
software  engineering  and  serve  as  introduction  to 
researchers 

University  of  York  (UK) 

Software  systems  with  a  high  requirement  for 
dependability. 
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Sample  Target  Student 


Air  Force  1  nstitute  of  Technology 

PMs  and  software  developers  from  DoD  &  other 
agencies 

California  State  University  -  Sacramento 

UG  degree  with  CS  courses 

Embry- Riddle  Aeronautical  University 

Strong  academic  record 

George  Mason  University 

UG  degree 

Monmouth  University 

UG  degree  in  CS,  SWE  or  Engineering  related 

Naval  Postgraduate  School 

Acquisition  professionals  with  2+  years  in  software 
development 

Seattle  University 

UG  degree  in  CS  or  equivalent  2+  years  in 
software  development 

Stevens  1  nstitute  of  Technology 

Experienced  computer  professionals  seeking 
leadership  positions 

University  of  Alabama  -  Huntsville 

UG  courses  in  CS,  math  &  statistics 

University  of  Southern  California 

UG  degree  in  CS,  math  or  engineering  with 
courses  in  computing  and  math 

University  of  York  (UK) 

UG  degree  in  CS  or  related  field  with  math 
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I  ntroduction  to  Software  Engineering 


AFIT 

Air  Force  1  nstitute  of  Technology 

CSE  481 

CSUS 

California  State  University  -  Sacramento 

ERAU 

Embry- Riddle  Aeronautical  University 

GMU 

George  Mason  University 

- 

MMU 

Monmouth  University 

SE  504 

NPS 

Naval  Postgraduate  School 

SW  3460 

SEA 

Seattle  University 

SIT 

Stevens  1  nstitute  of  Technology 

CS  540 

UAL 

University  of  Alabama  -  Huntsville 

CS  650 

use 

University  of  Southern  California 

CS  577a,  CS  577b 

YOR 

University  of  York  (UK) 

- 

REQ 

Software  Requirements 

DES 

Software  Design 

CST 

Software  Construction 

TST 

Software  Testing 

MNT 

Software  Maintenance 

CNF 

Software  Configuration  Management 

MGT 

Software  Engineering  Management 

PRC 

Software  Engineering  Process 

TLS 

Software  Engineering  Tools  and  Methods 

QLY 

Software  Quality 

■2 

re 

u 

in 


1 .00  >  90%  of  subtopics 
0.75  ~75%  of  subtopics 
0.50  ~50%  of  subtopics 
0.25  ~25%  of  subtopics 
0.00  No  Coverage 
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I  ntroduction  to  Software  Engineering 


Air  Force  Institute  of 
Technology 


California  State 
University  -  Sacramento 


Embry  Riddle 
Aeronautical  University 


CST 


TST 


CNF 


Naval  Postgraduate 
School 


CNF 


Stevens  institute  of 
Technology 


CNF 


University  of  Alabama  - 
Huntsville 


CST 


TST 


1.00  >90%  of  subtopics  0.75  ~75%  of  subtopics  0.50  ~50%  of  subtopics 
0.25  ~25%  of  subtopics  0.00  No  Coverage 


Monmouth  University 

REO 


CNF 


University  of  Southern 
California 

REO 
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SWEBOK  Schools 


Required  and  Semi- Required  Courses 


AFIT 

Air  Force  1  nstitute  of  Technology 

CSUS 

California  State  University  -  Sacramento 

ERAU 

Embry- Riddle  Aeronautical  University 

GMU 

George  Mason  University 

MMU 

Monmouth  University 

NPS 

Naval  Postgraduate  School 

SEA 

Seattle  University 

SIT 

Stevens  1  nstitute  of  Technology 

UAL 

University  of  Alabama  -  Huntsville 

use 

University  of  Southern  California 

YOR 

University  of  York  (UK) 

REQ 

Software  Requirements 

DES 

Software  Design 

CST 

Software  Construction 

TST 

Software  Testing 

MNT 

Software  Maintenance 

CNF 

Software  Configuration  Management 

MGT 

Software  Engineering  Management 

PRC 

Software  Engineering  Process 

TLS 

Software  Engineering  Tools  and  Methods 

QLY 

Software  Quality 

3.00  >lReq.  or  Semi  Req.  Course 
2.00  1  Req.  or  Semi  Req.  Course 
1.00  I  ntroductory  course 
0.00  No  Course 
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Air  Force  Institute  of 
Technology 


TL8 


PRC 


CNF 


Monmouth  University 

REO 


TL8 


C8T 


PRC 


California  State 
University  -  Sacramento 

REO 


TL8 


C8T 


PRC 


T8T 


Naval  Postgraduate 
School 

REO 


TLS 


PRC 


CNF 


CST 


TST 


University  of  Alabama  - 
Huntsville 

REO 


TL8 


C8T 


PRC 


T8T 


University  of  Southern 
California 

REO 


PRC 


TLS 


C8T 


CNF 


Embry  Riddle 


CNF 


George  Mason  University 


TL8 


PRC 


CNF 


Seattle  University 

REO 


CNF 

University  of  York 

REO 


Stevens  I  nstltute  of 
Technology 

REO 


CNF 


3.00  >lReq.  or  Semi  Req.  Course 
2.00  1  Req.  or  Semi  Req.  Course 
1.00  I  ntroductory  course 
0.00  No  Course 


Required  and 
Semi- Required 
Courses 


Non-SWEBOK 


Non-SWEBOK 


1 

oo 

Object  Oriented  Systems 

2 

Ul 

User  Interface  /  human  computer  interaction 

3 

RM 

Research  Methodology 

4 

SE 

Systems  Engineering 

5 

EMB 

Embedded  &  realtime  software  systems 

6 

RE 

Software  Reliability 

7 

DIS 

Distributed  Software  Engineering 

8 

ECO 

Software  Engineering  Economics 

9 

WWW 

SwE  for  worldwide  web 

10 

SEC 

Software  Safety  &  Security 

11 

MAT 

Math  foundations  of  SwE 

12 

PRO 

Programming 

13 

ALG 

Algorithms 

14 

DBS 

Database  Systems 

15 

SOA 

Service  Oriented  Architecture 

16 

OS 

Operating  Systems 

17 

ARC 

Computer  Architecture 

18 

RD 

Software  R&D 

3.00 

2.00 

1.00 

0.00 


(Required  and  Semi-Required  Courses) 


More  than  1  full  course 
Full  Course 
Partial  Course 
No  Course 
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Air  Force  Institute  of 
Technology 


oo 


SEC 


Monmouth  University 


SEC 


California  State  University 
Sacramento 


SEC 

Naval  Postgraduate 

School 

oo 

RD.  3— t-  Ul 


SEC 


University  of  Alabama  - 
Huntsville 


University  of  Southern 


SEC 


SEC 


Embry  Riddle 
Aeronautical  University 


SEC 


Seattle  University 


SEC 


George  Mason  University 


oo 


SEC 

Stevens  Institute  of 


U  niversity  of  York 

oo 

RD  3-- 1 — .  Ul 


SEC 


3.00  More  than  1  full  course 
2.00  Full  Course 
1.00  Partial  Course 
0.00  No  Course 


Non-SWEBOK 
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Non-SWEBOK 


Cumulative  Coverage 
oo 


MODE 

No.  of  Schools  (Max  11) 


MAT 


SEC 
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Early  Start  Team  Members 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 
11. 
12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 
21. 
22. 


Bruce  Amato,  Department  of  Defense 

Mark  Ardis,  RIT 

Larry  Bernstein,  Stevens 

Barry  Boehm,  USC 

John  Brackett,  Boston  University 

Murray  Cantor,  IBM 

Robert  Edson,  ANSER 

Gary  Hafen,  NDIA  and  Lockheed  Martin 

Tom  Hilburn,  Embry-Riddle  Aeronautical  University 

Jim  McDonald,  Monmouth  University 

Ernest  McDuffie,  National  Coordinating  Office 

Bret  Michael,  NPS 

Bill  Milam,  Ford 

Ken  Nidiffer,  SEI 

Art  Pyster,  Stevens 

Paul  Robitaille,  INCOSE  and  Lockheed  Martin 
Doug  Schmidt,  Vanderbilt 
Mary  Shaw,  Carnegie  Mellon  University 
Richard  Thayer,  California  State  University  at  Sacramento 
Richard  Turner,  Stevens 
Osmo  Vikman,  Nokia 
David  Weiss,  Avaya 


Graduate  Students: 
•Deva  Henry 
•Kahina  Lasfer 
•Sarah  Sheard 


Observer: 

•Joe  Urban,  NSF 
•Lillian  Cassel,  ACM 


Several  more  offers  and  lots  of 
interest... 
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Status  -  Create  Strawman  Curriculum 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  a  strawman  model  curriculum,  suitable  for  broad 
use,  with  a  small  representative  team  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 

etc.  (continuous) 

4.  Gradually  obtain  endorsement  from  ACM,  IEEE, 
INCOSE,  NDIA,  and  other  professional  organizations 

(continuous) 

5.  Create  full  model  curriculum,  suitable  for  global  use,  with 

a  large  representative  team  (September  2008  and  September 
2009) 

6.  Seek  early  adopters  (continuous) 
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Creating  the  Strawman  Curriculum 


1 .  Held  workshop  on  August  15-16  at  Applied  Systems  Thinking 
Institute 

2.  Reviewed  foundational  documents:  SEI  graduate  curriculum 
reports  from  1991,  SWEBOK,  SE2004,  INCOSE  SE  Model 
Graduate  Curriculum 

3.  Agreed  to  create  strawman  curriculum  and  agreed  on  outline 
of  document 

4.  Divided  into  4  primary  teams  with  leads  from  4  different 
universities 

•  Guidance  and  Outcomes  -  Art  Pyster,  Stevens  Institute 

•  Curriculum  Architecture  -  Jim  MacDonald,  Monmouth 

•  Body  of  Knowledge  -  Tom  Hilburn,  Embry-Riddle 

•  Course  Packaging  -  Brett  Michael,  Naval  Postgraduate 
School 

5.  Agreed  to  work  in  parallel  where  possible  to  speed  delivery^ 


Creating  the  Strawman  Curriculum 


6.  Build  Guidance  and  Outcomes  as  deltas  from  SE2004 
Principles  (Draft  1  done) 

7.  Build  Architecture  starting  with  1991  SEI  curriculum 
architecture  (Draft  1  under  review) 

8.  Build  Body  of  Knowledge  as  deltas  from  SWEBOK  using 
INCOSE  Handbook,  PMI  BOK,  and  current  state  of  SWE 
graduate  programs  as  primary  sources  for  additions  (Draft  1 
begun) 

9.  Build  Course  Packaging  after  first  three  teams  have  solid 
drafts 

10.  Hold  second  workshop  in  December  to  review  progress 

1 1 .  Refine  drafts  and  publish  at  end  of  February 
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Sample  Draft  Guidance 


Software  Engineering  draws  its  foundations  from  a  wide  variety  of  disciplines. 

•  Graduate  study  of  software  engineering  relies  on  many  areas  in  computer  science 
for  its  theoretical  and  conceptual  foundations,  but  it  also  draws  from  other  fields, 
including  statistics,  logic,  calculus,  discrete  mathematics,  formal  languages,  and 
other  mathematical  specialties,  from  systems  and  domain  engineering,  from 
project  and  portfolio  management,  and  from  one  or  more  application  domains. 

MSwE2008  must  identify  prerequisite  requirements  for  students  to  enter  an 
MSE  program. 

•  Undergraduate  computing  programs  and  industry  experience  in  software 
engineering  vary  greatly.  To  help  institutions  build  programs  that  address  the 
needs  of  the  broad  software  engineering  community,  MSwE2008  recommends 
minimum  prerequisite  knowledge  necessary  to  successfully  engage  in  a  program 
based  on  the  MSwE2008  curriculum.  Generally,  that  knowledge  comes  from  a 
technical,  scientific,  or  engineering  undergraduate  degree  including  coursework  in 
computer  science.  However,  relevant  work  experience  can  substitute  for  formal 
education.  Schools  that  wish  to  admit  students  lacking  that  minimum  prerequisite 
knowledge  should  provide  preparatory  courses  that  those  students  should  take 
before  entering  the  Masters  program. 
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Sample  Draft  Outcome 


Show  mastery  of  the  software  engineering  knowledge  and  skills ,  and 
professional  issues  necessary  to  practice  as  a  software  engineer  in  a  variety  of 
application  domains  with  demonstrated  performance  in  at  least  one 
application  domain. 

Students,  through  regular  reinforcement  and  practice,  need  to  gain  confidence  in  their 
abilities  as  they  progress  through  a  software  engineering  program  of  study.  At  graduation,  a 
student  should  understand  what  distinguishes  practice  in  different  application  domains  such 
as  finance,  medical,  transportation,  and  telecommunications,  should  understand  how  to  learn 
a  new  domain  as  needed,  and  should  demonstrate  skill  as  a  software  engineer  in  at  least  one 
application  domain.  Such  demonstration  will  include  (as  defined  in  Bloom’s  Taxonomy) 

—  At  least  comprehension  level  competency  across  all  MSwE2008  BOK  knowledge 
areas,  not  including  the  KA  on  “Knowledge  Areas  of  the  Related  Disciplines”. 

-  Application  level  competency,  or  above,  in  75%  of  the  MSwE2008  BOK 
knowledge  areas. 

Hence,  a  graduate  should  be  able  to  analyze,  design,  verify,  validate,  implement,  apply,  and 
maintain  a  modest-sized  software  system  and  understand  the  challenges  of  scaling  to  larger 
software  systems.  In  addition,  graduates  need  to  have  gained  an  understanding  and 
appreciation  of  professional  issues  related  to  ethics  and  professional  conduct,  economics, 
and  the  societal  needs. 
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Sample  Draft  Outcome 


Work  effectively  as  part  of  a  team ,  including  teams  that  may  be 
international  and  geographically  distributed '  to  develop  quality 
software  artifacts,  and  to  lead  in  one  area  of  project  development, 
such  as  project  management,  requirements  analysis,  architecture, 
construction,  or  quality  assurance. 

Students  need  to  complete  tasks  that  involve  work  as  an  individual,  but  also 
many  other  tasks  that  entail  working  with  a  group  of  individuals.  For  group 
work,  students  ought  to  be  informed  of  the  nature  of  groups  and  of  group 
activities/roles  as  explicitly  as  possible.  This  must  include  an  emphasis  on 
the  importance  of  such  matters  as  a  disciplined  approach,  the  need  to  adhere 
to  deadlines,  communication,  and  individual  as  well  as  team  performance 
evaluations.  Students  should  have  an  appreciation  of  team  dynamics  and 
leadership  techniques  and  be  able  to  lead  at  least  one  of  the  areas. 
Increasingly,  teams  are  assembled  from  many  geographical  sites,  often 
across  national  boundaries.  This  presents  additional  challenges  of  time, 
language,  and  culture  that  students  must  know  how  to  address. 
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Status  -  Obtain  Endorsement 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  a  strawman  model  curriculum,  suitable  for  broad 
use,  with  a  small  representative  team  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 

etc.  (continuous) 

4.  Gradually  obtain  endorsement  from  ACM,  IEEE, 
INCOSE,  NDIA,  and  other  professional  organizations 

(continuous) 

5.  Create  full  model  curriculum,  suitable  for  global  use,  with 

a  large  representative  team  (September  2008  and  September 
2009) 

6.  Seek  early  adopters  (continuous) 
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Endorsements 


•  NDIA  SE  Division  endorsed  iSSEc  in  June  2007 

•  INCOSE  Board  of  Directors  endorsed  iSSEc  in  October  2007 

•  ACM  Education  Board  is  following  iSSEc  progress  and  is 
considering  endorsement 

•  IEEE  Computer  Society  is  following  iSSEc  progress  and  is 
considering  endorsement 

•  Endorsement  from  other  organizations  is  possible 
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Finally... 


•  The  team  working  on  the  Strawman  Curriculum  has  been 
doing  a  great  job  and  are  keeping  to  the  planned  schedule 

•  A  workshop  among  the  broad  community  to  review  the 
Strawman  Curriculum  and  to  plan  the  creation  of  the  full 
curriculum  will  be  held  in  March  or  April  2008  -  hope  to 
publish  another  iteration  in  September  2008  and  another  in 
September  2009  that  reflects  broad  community  involvement 

•  Expect  a  number  of  early  adopters,  including  schools 
represented  on  the  Early  Start  Team  that  is  building  the 
Strawman  Curriculum 

•  Ultimately,  iSSEc  may  create  a  model  curriculum  for  an 
interdisciplinary  degree  that  fully  integrates  software  and 
systems  engineering  graduate  education 


James  I.  Finley 

Deputy  Under  Secretary  of  Defense 
(Acquisition  &  Technology) 

October  23,  2007 


The  Honorable 


NDIA  Conference  on  Systems  Engineering 


A&T  Vision 


LEADERSHIP 

for  an 

INTEGRATED,  RESPONSIVE 
ACQUISITION  SYSTEM 

providing 

WARFIGHTER  NEEDS 

with 

PREDICTABLE  PERFORMANCE 

“ THE  WILL  TO  CHANGE  ..." 
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Strategy 


RESHAPE  THE  ENTERPRISE 

UTI  LI  Zl  NG  SHORT  AND  LONG  TERM 

INITIATIVES 

THAT 

ACCELERA  TE  LASTING  CHANGE 

FOR  ALL  ELEMENTS  OF  THE 

ACQUISI 77  ON  SYSTEM 
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Goals 


•  Communication 


•  Cycle  Time 

*  Competitiveness 
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Impact  of  Not  Starting  Programs  Right 


My  observations  since  last  year... 

•  Programs  usually  fail  because  we  don't  start  them  right: 

-  Requirements  instability/ creep  -  not  well  defined,  not  understood 

-  I  nadequate  early  technical  planning 

-  I  nadequate  funding  or  phasing  of  funding  to  properly  execute  the 
program 

-  Lack  of  schedule  realism  -  success  oriented,  concurrent,  poor 
est i  mat i  o n/ pi  a  n  n  i  ng 

-  Lack  of  technical  maturity  or  a  credible  back-up  plan  -  “we're  always 
optimistic" 

-  Limited  focus  on  life  cycle  issues 


Program  success  depends  on  rigorous,  thorough,  technical 

planning  and  supportive  resources 
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Balanced,  Early  Life  Cycle  Planning 


•  Acquisition  strategy  -  realistic,  effective,  and  executable 

•  Cost  estimate  -  accurate 

•  I  ntegrated  technical  planning  (SE  /  T&E  /  SW  /  'ilities) 

•  Technology  identification  and  maturity 

•  Supportive  business  rules  (RFP,  contract,  etc.) 

•  Entrance  /  Exit  criteria  at  each  milestone 

•  Risk  identification  /  mitigation 

•  I  ncreased  Competition  and  Prototyping 

Requires  disciplined  leadership  to  stick  with  the  plan 
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What  we  need  from  you... 

•  Tell  your  leadership  that  Dr.  McQueary  and 
Dr.  Finley  are  focused  on  starting  programs 
right! 

•  We  are  working  daily  to  improve  communication, 
both  in  DoD  and  with  I  ndustry 

•  We  are  looking  to  improve  competition  and  time 
to  field  capabilities 
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We  owe  them  our  very  best!!! 
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Headquarters  U.S.  Air  Force 

Integrity  -  Service  -  Excellence 

Status  of  USAF  Systems 

Engineering 

Mr.  Terry  Jaggers,  SES 
Deputy  Assistant  Secretary 
Science,  Technology,  &  Engineering 


23  Oct  2007 


What  We’ve  Done 


■  Published  AFI  63-1201,  Life  Cycle  Systems  Engineering  (LOSE)  --  first 
AF  policy  for  SE 

■  Published  AFMCI  63-1201,  OSS&E  and  Systems  Engineering  Life  Cycle 
Management 

■  Life  Cycle  SE  initiatives  pushed  into  all  AFS021  initiatives  for  which  SE 
is  a  key  enabling  process 

■  Created  Unit  Compliance  Inspection  (UCI)  checklist  for  SE  at  AFMC 

■  Creating  integrated  Weapon  System  Integrity  Program  to  ensure 
cohesive  SE  effort  with  all  integrity  programs 

■  Developing  AFMC-wide  Risk  Management  Tool  pilot 

■  Begun  insertion  of  SE  practices  into  Probability  of  Program  Success 
(PoPS)  management  tool 

■  In  final  steps  of  ensuring  100%  SEP  compliance  across  all  air  and 
space  ACAT  programs  to  meet  SECAF  direction 

■  Increased  workforce  participation  in  SE  graduate  education  and 
distance  learning  programs 

■  Completed  National  Research  Council  (NRC)  study  on  early  SE 

■  Funded  SMC  pilot  project  to  develop  and  validate  process 
documentation  --  report  ECD  Nov  07 
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Where  We're  Going 


■  Developing  a  Corporate  AF  SE  Assessment  Module  to  support  core 
SE  attributes  across  both  AFSPC  and  AFMC  -  process  areas,  process 
area  goals,  practices,  and  evidence 

■  Including  an  AFIT  SE  Masters’  Degree  program  in  Civilian 
Developmental  Education 

■  Investigating  governance  framework  for  enterprise  architecting  and 
system-of-systems  (SoS)  engineering  at  CSE 

■  Increasing  academic  research  (in-house  &  collaborative)  at  AFIT  CSE 

■  Enhancing  integration  of  related  specialty  areas  (software,  HSI, 
manufacturing,  etc.)  for  inclusion  in  increment  2  of  AFI  63-1201 

■  Establishing  an  AFMC  knowledge  management  toolset  or  forum  to 
assist  programs  with  issues  in  planning  or  executing  SE 

■  Preparing  a  study  to  review  best  practices  from  TRAs,  MRAs,  IPAs, 
PSRs,  IRTs,  etc.  as  a  Corporate  AF  gold  standard  for  deep  dive  tech 
planning  reviews 

■  Refining  policy,  processes,  programs  and  people  issues  to  implement 
early  SE  under  the  Life  Cycle  Systems  Engineering  (LCSE)  construct 
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What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 


What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 


What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 


Traditional  Life  Cycle  SE  Definition 


6 


What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 


Traditional  Life  Cycle  SE  Definition 


What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 
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Traditional  Life  Cycle  SE  Definition 
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What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 
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Translate 
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needs  into  a  set  of 
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describing  a 
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“Early  SE”  Traditional  Life  Cycle  SE  Definition 


What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 
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Traditional  Life  Cycle  SE  Definition 
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What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 
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What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 
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“Early  SE” 


Traditional  Life  Cycle  SE  Definition 


LCSE  Defined  by  Air  Force  Instruction  (AFI  63-1201) 
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What  is  AF  Life  Cycle  Systems 

Engineering  (LOSE)? 


F  EQUIREMENT 


fc—f 


YSTEM  ANALYSIS 


Translate 


CONTROL 


j  needs  into  a  set  of 

M(.  , 

requirements 
describing  a 
conceptual 


□  Like  a  SEP,  the  SE  process  during 
the  LCSE-C  phase  is  governed  by  a 
“concept  engineering  plan”  or  a 
(ConSEP) 

□  Like  the  System  Design  Spec,  the 
product  of  the  LCSE-C  phase  is  a 
concept  design  or  a  “concept 
characterization” 

□  Like  verification  of  the  system 
design,  the  concept  characterization 
needs  verification  &  traceability 


Early  SE  leads  to  better  military  utility  assessments  &  better  life  cycle  cost 
estimates,  which  inform  decisions  &  ultimately  lowers  risk  during  acquisition 
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Early  SE  Challenges 


Policy 

■  Establishing  formal 
milestones  earlier 

■  Establishing  criteria  to 
measure  early  SE  products 
at  these  milestones 

Programs 

■  Institutionalizing  funding  for 
consistent  SE  application 

■  Placing  early  concept  SE 
products  under 
configuration  control 


Process 

■  Developing  ConSEP  guides 

■  Templates  for  Concept 
Characterization  documents 

■  Capturing  SE  content  in  IT  to 
move  forward  with  programs 

People 

■  Identifying  early  SE  expertise 
&  “systems  thinkers” 

■  Ensuring  the  right  balance 
between  engineers  & 
analysts,  military  &  civilian 
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Summary 


■  AF  Making  Progress  in  SE  Revitalization 


■  AF  Will  be  Resource  Constrained  in  the  Future 


■  Early  SE  Provides  Opportunities  to  the  AF 


Pursuing  USAF  Technical  Excellence ! 
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Special  Thanks 


Col  Jim  Horejsi 
Col  Rich  White 
Maj  David  Borgeson 
1st  Lt  Steve  Dirks 
and  the  SMC  Team 

THE  AEROSPACE 

^CORPORATION 


INNOVATIVE  TECHNOLOGY  SYSTEMS 


Great  Work  on  the  Pre-A  SE  Process  (PASEP)  Study 
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Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


A  Test  Strategy  Done  Early  Drives  Test  Planning 
and  Successful  Testing 


National  Defense  Industrial  Association  (NDIA) 
10th  Annual  Systems  Engineering  Conference 
October  22-25,2007 

Test  &  Evaluation  in  Systems  Engineering  Track, 
Tuesday  October  23,  2007 


William  Lyders,  ASSETT  Inc. 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


Agenda 


□  T&E  Strategy,  an  important  early  document  in  the  SE 
Process 

□  Key  Information:  Contents  in  a  T&E  Strategy 

□  Where  the  Features  of  a  Test  Strategy  are  Verified 

□  T&E  Lessons  Learned  at  ASSETT 

□  Summary  and  Conclusions 

□  Q&A 


23  October  2007 
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A  Test  Strategy  Sets  the  Stage  for  Testing 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


□ 


□ 

□ 

□ 


Test  Strategy  is  a  high  level 
description  of  major  system 
wide  activities  that  achieve 
the  project's  testing 
objectives 

It  outlines  the  approach  to 
ensure  the  system  is 
adequately  tested 

Ground  rules  for  writing  the 
Test  Plan 

Done  early  in  the  life  cycle, 

it  is  a  generic  approach  and 
defines  the  basis  for  test 
plans  and  test  procedures 
to  follow. 

■  Sometimes  features 
done  with  proposal 

■  Should  complete  by  PDR 


One  early  Strategy  is  to  define  the  Test 
Environment  Locations  for  Test  Events 


23  October  2007 
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SSEYY  T&E  Strategy,  an  early  SE  Process  product 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


The  features  of  a  Test  Strategy  should  all  be  baselined  by  the  PDR 
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System 
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Data 
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Level  Design 
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Requirements 

Traceability 

Verification 
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Test  Plans 
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Test 

Traceability 

Matrix 


Release 

Content 
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Production 
Plan 
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Plans 


SE 


T&E 


23  October  2007 
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Key  Information:  Contents  in  a  Test  Strategy 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


Table  of  Contents 

□  Introduction 

□  Scope 

□  Testing  Objectives 

□  Assumptions 

□  A  Risk  Assessment 

□  The  Critical  Attributes  as  Test  Focus  Areas 

□  The  Levels  of  Testing  to  be  Included 

□  Types  of  Tests 

□  Test  Methods 

□  Organizational  Responsibilities  for  Testing 

□  Entry/Exit  Criteria 

□  Test  Equipment 

□  Test  Metrics  relevant  to  Quality  Criteria 

□  Test  Management  and  Reporting 

□  Approvals 

□  Glossary  of  Terms 


23  October  2007 
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Contents:  Test  Scope  &  Objectives 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


Bounding  the  amount  of  Testing  early  is  a  key  test  program  planning  objective 


Test  Scope 


•  Define  boundaries  of  test  - 
a  test  work  flow 

•  Contractual  limitations 


•  Lists  of  applicable  system 
components  planned  in  tests 

•  Emphasis  on  a  cost 
effective,  team  approach 


•  SOW  emphasis  on  specific 
efforts  to  verify  technical 
requirements  and  limit  risky 


Testing  Objectives 


•  Clearly  define  overall 
project  technical  objectives 

•Define  the  multiple  levels  of 
testing  within  the  test  scope 

•  Laboratory 

•  Field  Test 

•  Test  Platform 

•  Operational 


•Technical  objectives  for  each 
level  of  testing 


•Customer  objectives  and 
importance 


23  October  2007 
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Contents:  Assumptions 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


The  key  and  critical 
underlying  assumptions 

\  For 

For  your 

^your 

behind  test  strategy 

team 

y/  efforts 

customer^ 

Test  Environments 

•  Locations  of  different  test  environments 

•  Infrastructure  requirements 

•  Simulation/Stimulation  fidelity 

•  Development  and  T&I  environment  HW/SW 

•  Owners  and  maintenance  responsibilities 

•  Sharing  of  components  between  environments 


Test  Operations 


•  Company/organizations  testing  each  component 

•  Hardware/ Software  delivery  schedule  and  methods 

•  Planning  and  conducting  DT&  E  (T&I)  and  OT&E 

•  Plans  for  incorporation  of  OT&E  early  in  cycle 

•  Test  teams  by  company  and  test  event 

•  Field  testing  and  Customer  site  acceptance  tests 

•  Discrepancy  level  definitions  -  PTR  Severity _ 

•  Regression  testing 


Test 

Documentation 

•Information  Level 
&  Product  Schedule 

•  Components 

•  System  T&I 

•  Test  Plans 

•  Test  Procedures 

•  Test  Reports 
•  Test  Data 


23  October  2007 
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Contents:  Risks  and  Critical  Attributes 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


Risks  to  successful  testing  should  be  assessed  with  mitigation  plans 


Risk 

# 

Risk  Description 

Probability 

(H,  M,  L) 

Priority 

(H,  L,  M) 

Mitigation 

1 

IF  the  Cabinet  Components  for 
the  First  Article  TPS  Cabinet  do 
not  arrive  or  are  not  accepted  per 
the  IVT  start  date  THEN  the 
Cabinet  IVT  will  not  be  able  to 
start  on  time. 

M 

M 

The  Cabinet  components  ordered  for  the 
Prototype  required  a  longer  lead  time  than 
originally  expected.  So  the  FA  components  are 
being  ordered  earlier  than  originally  planned. 

2 

IF  the  Cabinet  Integration  is  not 
successfully  completed  by 

January  4,  2008  so  the 

Certification  Test  can  be 
completed  by  22  January,  2008  ^ 
THEN  the  Cabinet  cannot  be 
shipped  to  the  Prime  and  their  \ 
Integration  of  the  Cabinet  will  be 
impacted. 

M 

# 

ASSETT  is  investigating  advancing  the 
component  purchasing  plans  to  be  able  to  have 
the  Cabinet  components  1-2  months  earlier. 

This  could  allow  a  1-2  month  earlier  start  for 
Cabinet  Integration. 

Also  the  planned  test  time  for  IVT  includes 
some  buffer  time  for  unexpected  problems. 

Test  Focus  areas  or  the  Critical  Attributes  of  the  system  that  must  be  tested  to 
provide  the  customer’s  level  of  confidence  in  the  system 

Examples  of  critical  attributes: 

•  System  thru  put 

•  Ability  to  handle  specific  data  types 

•  Ability  to  support  legacy  interfaces 

•  Ability  to  be  maintained 

•  Reliability  if  a  very  high  reliability  is  necessary 

•  Performance 

•  Support  operational  situations 
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Contents:  Plan  Strategies  for  each  Level  of  Testing 
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The  System  Integration  Planning  Strategy  Must  Address  All  Levels  of  Test  & 
Leverage  the  Coverage  at  Each  Level  to  Eliminate  Duplication 


Integration 

Planning 

’  System  1 

Subsystem  1 

Platform 

— 

NCO  =>  System  of  Systems 

System  N 

Subsystem  N 

□  Levels  of  Planning 


■  Platform 

■  System 

■  Subsystem 

■  Unit/Program 

■  Configuration  Item 

■  Component 

■  Field  Test 

□  Results  in  Multiple  Test  Plans 


System  Test  Strategy 

Acceptance  Criteria  for  each  system 
requirement  is  identified. 

Test  Strategy  for  each  system  requirement  is 
established. 
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Example  Test  Level  Strategy  Work  Flow 
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Create  a  flow  diagram  of  planned  test  activities  to  visualize  schedule  &  overlaps 

Legend: 


Critical  Item  Testing 


Part  of  Integration  Verification  Testing  (IVT) 


Thermal  Performance 


Custom  Components 
Acceptance  Verification 


Purchase  Specifications 

COTS  Components 
Purchase  Acceptance 


Purchase  Specification 


Simulators 

•  Combat  System 

•  Arrays 


Subsystem 


Integration 
Testing  of 
Assett  Cabinet 

•  Design  Verification 

•  Requirements 

•  Cabinet  Spec 

•  All  HW  &  FW 


Integration 

Test 

Plan/ 

Procedure^ 


FW  Testing  j  Component 


j  FW  Testing 

Commercial  Equipment  Test  Bed 


Testing  w/2nd  Unit 


•  Customer  Testing  @ 

^  ^“Their  Site 

•  3  weeks  Elapsed  Time 

•  ASSETT  Support 


Customer 
Certification 
Testing 

•  Customer  Test  @  ASSETT 

•  ASSETT  Support 

•  2  Weeks  Elapsed  Time 

•  Operational  Capabilities 

•  No  Limit  Testing 

•  Test  Specification 


1  \ 

Requirements 

Verification 

Matrix  (RVM) 

_ 
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Contents:  Types  of  Testing  and  Strategies 
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The  Types  of  Testing  can  be  planned  for  by  the  types  of  requirements 


•Test  Planning  by  Types 

•  Functional: 

•  Business  Functions,  System  Functions, 

•  System  Performance,  Interfaces,  etc. 

•  Non-Functional: 

•  Construction,  Shipping,  Environmental,  RMA,  etc. 

•  Schedule  Driven  vs.  Event  Driven  Strategies 

•  Event  Driven  Testing 

•  Event  driven  best  for  reducing  technical  risks 

•  Only  advance  when  lower  level  of  technical  verifications  are  completed 

•  Component,  Unit,  Cabinet  or  Computer  Program,  ....  System 

•  Can  be  more  expensive  if  delays  in  Subsystem  or  System  level  testing  impacted 

•  Schedule  Driven  Testing 

•  Rigorous  schedule  of  testing  to  consider  funding  profile 

•  Integrating  DT&E  Events  with  early  OT&E  Events 

•  DT&E  is  Laboratory  level  testing  -  little  or  no  human  environment 

•  DT&E  uses  simulators  for  operating  environment  conditions 

•  OT&E  includes  the  human  element  in  testing 

•  OT&E  has  the  real  platforms  and  real  operating  environments 

•  Early  OT&E  Alignment  in  DT&E  environment  can  be  done 

•  Plan  long  duration  operability  demonstration  tests  with  real  system  operators 

•  Schedule  regular  test  shifts  for  3-6  months  for  real  system  operators 
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Contents:  A  Test  Procedure  Strategy 
and  Test  Methods 
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Strategy:  Consider  leveraging  early  requirement  verifications  for  integration  testing 


Strategies  for  which  Test  Methods  to  later  be  defined  for  each  requirement  or  use  case 
need  to  be  identified  early 


•  Analysis  (A)  -  Verification  by  analysis  includes  quantitative  proof  that  an  item  meets  the  requirement  by  the  technical 
evaluation  of  equations,  diagrams,  or  representative  data. 

•  Demonstration  (D)  -  Verification  by  demonstration  includes  exercising  an  item  to  provide  qualitative  data  that  an 
item  meets  a  requirement. 

•  Hierarchical  (H)  -  These  requirements  contain  no  significant  content  but  are  used  to  establish  a  hierarchical 
relationship  between  requirements.  They  are  verified  by  the  verification  of  their  subordinate  requirements.  (Does  not 
require  a  test) 

•  Inspection  (I)  -  Verification  by  inspection  includes  examination  of  hardware  or  documentation  to  see  if  an  item  meets 
the  requirement. 

•  Similarity  (S)  -  Verification  by  similarity  includes  a  comparison  of  one  item  to  another  item  that  has  been  verified  to 
meet  the  requirement. 

•  Test  (T)  -  Verification  by  test  includes  using  an  instrument  to  make  a  measurement  that  provides  qualitative  data  that 
an  item  meets  the  requirement. 
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Contents:  Organizational  Responsibilities,  Criteria 
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Consider  the  expected  Test  Organizations  and  their  Responsibilities  in  your  plans 


Role/Responsibility 

Name/Organization 

Phone  number 

ASSETT  Test  Director 

Bill  /ASSETT 

800-555-1234 

ASSETT  Hardware  Tester  #1 

Patrick  /ASSETT 

800-555-4567 

ASSETT  Hardware  Tester  #2 

Monty  /ASSETT 

800-555-6789 

ASSETT  Firmware  Tester  #1 

Rob  /ASSETT 

800-555-1357 

ASSETT  Firmware  Tester  #1 

Rod  ASSETT 

800-555-2468 

Customer  Witness  #1 

Witness  #1 /EDO 

Customer  Witness  #2 

Witness  #2/cuu 

Define  Specific  Success  Criteria  in  order  to  Enter  &  Exit  Test  Events  and  Levels 


•Entry  Criteria 

•  Establish  pre-requisites  before  starting  any  series  of  test  events 

•  Example:  testing  of  a  work  product  at  the  previous  level  completed 

•  Example:  No  significant  PTRs 
•Exit  Criteria 

•  Establish  post-conditions  before  declaring  completion  of  test  events 

•  Agree  on  Pass/Fail  Criteria,  e.g.  statistical  performance  testing;  reruns 

•  Example:  Customer  accepts  work  product 

•  Example:  Acceptable  level  of  each  open  Discrepancy  level  (PTRs) 
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Other  Contents 


(Page  1  of  2) 
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□  Test  Equipment 

■  Technology  needed  in  test  environments 

■  Determine  what  will  be  bought,  GFE,  CFE,  capitalized 

■  If  field  tests  involved,  determine  transportability 


□  Test  Metrics 

■  Metrics  programs  or  measurement  strategy 

■  Metrics  could  drive  test  method  scope  and  equipment  needed 


□  Test  Management  and  Reporting 

■  T&E  Manager,  Test  Director,  and  organization 

■  Approach  to  reporting  daily  &  final  test  results 

■  Capturing,  tracking  and  reporting  discrepancies  (PTRs)  against 
requirements 

■  Test  Reports 
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Other  Contents 
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(Page  2  of  2) 


□  Approvals 

■  How  are  tests  to  be  witnessed 

■  How  are  test  results  and  test  reports  approved 

■  What  constitutes  a  successful  completion  by  your  customer 


A  Glossary  of  abbreviations  for  the  project  is  very  helpful  in  understanding  terminology 


AIS 

Active  Intercept  Sonar 

BCR 

Baseline  Change  Request 

CAS 

Cylindrical  Array  Sonar 

CAT 

Component  Acceptance  Test 

CC&S 

Combat  Control  and  Surveillance 

Cl 

Critical  Item 

CIT 

Critical  Item  Testing 

CDR 

Critical  Design  Review 

CDRL 

Contract  Data  Requirements  List 

CMP 

Configuration  Management  Plan 

PCA  Physical  Configuration  Audit 

PCI  Production  Control  Inspection 

PI  Production  Inspection 

PRS  Passive  Ranging  Sonar 

POC  Point  of  Contact 

PTR  Project  Trouble  Report 

QA  Quality  Assurance 

QMP  Quality  Management  Plan 

RVDS  Requirements  Verification  Data  Sheet 

RVM  Requirements  Verification  Matrix - 
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Some  DT&E  &  OT&E  Lessons  Learned  at  ASSET! 
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□  Get  testable  requirements  and  test  pass/fail  criteria 
defined  early  and  agreed  upon  with  the  Customer 

□  Create  a  Test  Strategy  and  Master  Test  Plan 
■  Get  buy  in  by  all  parties  involved 

□  Prepare  Test  Plans  for  each  of  the  different  levels  of 
integration  and  conduct  peer  reviews  and  customer 
reviews  as  necessary  -  don't  want  surprises  at 
acceptance 

□  Create  a  SRVM  and  get  it  reviewed/approved 

□  Fully  dry  run  all  test  procedures 

□  Document  all  test  findings  and  share  them  with  both 
Customer  and  own  teams 
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Test  Strategy  Contents:  Agreement  & 
Verification  done  at  Different  SE  Milestones 


Proposal/  SRR 
Milestone 

PDR  Milestone 

CDR  Milestone 

TRR  Milestone 

Scope 

- 

- 

- 

Test  Objectives 

Test  Objectives 

- 

- 

Assumptions 

Assumptions 

- 

- 

Risk  Assessment 

Risk  Assessment 

- 

□  Proposal 

Critical  Attributes 

□  Test  Strategies 

Critical  Attributes 
Types  of  Tests 

Test  Methods 
Organization 

Types  of  Tests 

Test  Methods 
Organization 

Entry/ Exit  Criteria 
Test  Equipment 

Test  Management 
Test  Reporting 
Approvals 

□  System 

□  Test  Architecture 

□  Requirements 

□  Test  Plans 

Requirements 

Specification 

Traceability  (SRVM) 

□  Test  Procedures 

□  Test  Data 

□  Test  Traceability 
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Summary  &  Conclusions 
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1.  An  initial  Test  Strategy  should  be  completed  very  early  in 
the  SE  Process... often  in  the  proposal! 

2.  More  detailed  Test  Strategy  Contents  can  be  defined  and 
refined  in  later  SE  Phases  in  Test  Architectures,  Test 
Plans,  &  Test  Procedures 

3.  Test  Strategy  Features  are  agreed  upon  early  and  verified 
at  design  reviews  and  test  readiness  reviews  during  the 
SE  Process 


Systems  Engineering  provides  a  structured  approach  to  managing  the  technical  solution 
over  the  full  life  cycle  from  concept  to  deployment  to  retirement... 

...Test  and  Evaluation  complements  this  approach  with  support  for  defining  requirements 
and  integration  planning. ..and  conducting  many  levels  of  integration  tests  with  systems 
engineering  support  to  achieve  customer  acceptance  of  a  system... 
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Q&A 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


23  October  2007 


Slide  19 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


Abstract 


Successful  test  and  evaluation  (T&E)  starts  at  the  beginning  of  the  SE  process  by  defining  testable 
requirements  and  a  test  strategy  for  verifying  those  requirements.  A  Test  Strategy  is  a  high  level 
view  of  a  project’s  Test  Plan  and  the  necessary  Test  Equipment  Support.  The  SE  defines  the  test 
strategy,  implements  it  in  test  plans  &  test  procedures,  and  then  supports  the  T&E  team  doing  the 
testing.  This  presentation  will  identify  lessons  learned  and  how  ASSETT  Inc.  has  successfully 
defined  test  strategies  for  both  large  and  small  projects  for  military  systems  both  with/without 
integrated  commercial  components. 

The  T&E  Strategy  is  an  important  early  document  in  the  SE  Process:  The  T&E  Strategy  is  a  high 
level  description,  developed  prior  to  PDR,  of  major  system-wide  activities  to  achieve  the  testing 
objectives.  It  outlines  the  planned  approach  to  ensure  the  system  is  tested  adequately.  An  event- 
drive  test  program  is  recommended  over  a  schedule  driven  test  program  to  reduce  technical  risk.  A 
transition  of  developmental  tests  (DT)  with  operational  tests  (OT)  is  recommended. 

The  Key  information  in  the  T&E  Strategy:  A  Test  Strategy,  whether  in  a  stand-alone  document  or 
incorporated  into  a  project  Test  Plan,  defines  the  testing  objectives,  assumptions,  an  initial  project 
risk  assessment,  test  focus  areas,  the  different  levels  and  types  of  testing,  the  test  organization 
responsibilities,  entry/exit  criteria  for  the  different  testing  levels,  test  tools,  and  any  metrics  relevant 
to  project  quality  criteria.  A  brief  explanation  and  the  importance  of  each  of  these  will  be 
summarized. 

Where  the  Features  of  a  Test  Strategy  are  verified:  At  the  PDR  where  a  Test  Plan  is  presented,  the 
features  of  a  test  strategy  begin  to  be  verified.  The  plans  for  validation  of  stakeholder  requirements, 
test  time  and  resources,  and  planned  tests  and  accompanying  test  procedures  are  reviewed.  These 
features  are  again  reviewed  in  more  detail  at  the  CDR.  And  finally,  at  the  TRR  prior  to  starting  the 
test,  the  requirements  traceability  and  test  resources  (tools,  procedures,  limitations,  and  validity  of 
the  test  procedures)  are  confirmed. 
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Author  Biography 


William  Lyders 
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and  multiple  SBIR  projects  for  the  Navy.  He  was  also  the  Test  Team  Lead  on  large  commercial 
projects  for  both  domestic  and  international  financial  institutions.  Mr.  Lyders  is  currently  Lead 
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Operational  Test  and  Evaluation 
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IOT&E  Results:  Effective,  Suitable 
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DOT&E  Priorities 


1.  Improve  Suitability 

2.  Enhance  operational  realism  in  early  tests 

3.  Provide  timely  performance  information  to  the 
war  fighters  and  fielding  decision  makers 

4.  Facilitate  adequate  OT  resources 

5.  Ensure  DOT&E  personnel  are  well  trained  and 
prepared 
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1.  Used  star  cluster  ammo  can  mounted  to  the  TC’s  50  Cal  ammo  can  with  tie  down  straps. 

2.  Fabricated  feed  chute  from  old  road  sign  to  top  feed  ammo  across  the  top  of  the  turret 
and  to  help  prevent  stoppages. 

3.  Door  providing  access  to  the  coax  held  open  to  allow  ammo  to  feed  into  the  COAX. 


DoD  O&S  Costs 

Largest  Fraction  of  Life  Cycle  Costs 


Ground  Combat  Systems 


Rotary  Wing  Aircraft 


Surface  Ships 


*  Source:  OSD  CAIG 

Fighter  Aircraft 


"We  have  a  tendency  to  look  at  what  it  takes  to  get  a 
program  out  the  door.  We  don't  think  too  much  about 
what  the  life  cycle  [cost]  is.  It's  'Can  I  build  it?'  I  would 
like  us  all  to  be  mindful  of  what  it  costs  to  operate 
whatever  we  are  building  for  whatever  its  life  is  going  to 
be  because  I  have  to  pay  that  bill  every  single  year.“ 


(CNO,  ADM  Michael  G.  Mullen  in  an  interview  with  "Government 
Executive"  magazine  May  15,  2006) 


Logistics  Management  Institute  Study 

Reliability  Investment  Rol  5:1  to  128:1 


Reliability  Investment  and  Support  Cost  Reduction 


MTBx  Hours 

*  CASA  20-Year  Support  Cost 
(costs  in  FY03  M  $,  discounted  7%) 

Economics 
(FY03  M  $) 

Case 

Study 

Was 

Is 

Percent 

Change 

Was 

Is 

Percent 

Change 

Reliability 

Investment 

ROI 

Predator 

40 

77 

92.5% 

$1,463 

$576 

60.6% 

$39.1 

22.7:1 

Global  Hawk 

67.7 

120 

77.3% 

$2,547 

$1,958 

23.1% 

$121.9 

5:1 

FBCB2 

47 

364 

674.5% 

$13,060 

$1,880 

85.6% 

$87.4 

128:1 

*  CASA  -  Cost  Analysis  Strategy  Assessment 


•  Reliability  investments  resulted  in  20-yr  support  cost  reductions 
•  Returns-on-lnvestment  ranged  from  5:1  to  128:1 


Defense  Logistics  Agency  Metrics 

Reliability  Investment  Rol  15.5:1 


DLA  Reliability  Investment  Summary 

Service 

Investment  ($M) 

Estimated  10  year 
LC  Savinas  f$M) 

ROI 

Army 

14.1 

187.0 

13.3:1 

Navy 

9.7 

207.0 

21.3:1 

Air  Force 

8.3 

102.0 

12.3:1 

Total 

32.1 

496.0 

15.5:1 

Source:  Karron  Small,  Aviation  Engineering  Directorate,  Defense  Supply  Center  Richmond,  Nov  2005 

Returns-on-lnvestment  in  reliability  15.5:1 


Empirical  Relationship 

Reliability  Investment  &  Reliability  Growth 
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Areas  That  Need  Engagement  of  SE  Community 


1.  Requirements 

-  Need  to  be  fully  understood  so  that  a  good  reliability  program  can  be  characterized 

-  Government  must  articulate  the  reliability  requirements  in  RFPs  or  Request  for 
Proposals,  to  provide  both  the  incentive  and  understanding  for  industry 

-  Industry  must  propose  appropriate  solutions;  and  then  systems  engineers  and 
management  must  follow  through  with  an  appropriate  design 

2.  Reliability  Objectives 

-  Need  to  be  an  integral  part  of  the  business  strategy  and  have  the  commitment  of 
senior  management  -  and  appropriate  incentives  must  be  in  place 

3.  Reliability  tasks 

Must  be  an  integral  part  of  SE  and  addressed  early  in  the  design  phase 

4.  Operational-use-environment  Duty  cycles.  Related  stresses 

-  Must  be  understood  for  the  entire  life  cycle 

5.  Root  cause  analysis  of  critical  failure  modes 

-  Must  be  done  to  eliminate  or  minimize  their  consequences 

6.  Reliability  of  the  design 

-  Should  be  verified  by  testing  and  analysis  to  ensure  requirements  are  achieved 


Actions  to  Improve  Suitability 

1.  Key  Performance  Parameter  (KPP)  for  Materiel  Availability  with  two  Key 
System  Attributes  (KSA)  -  Materiel  Reliability  and  Ownership  Costs 

2.  Sustainment  Metrics  Rationale  Report 

3.  DoD  Guide  for  Achieving  Reliability,  Availability,  and  Maintainability 

4.  Four  new  metrics  for  acquisition  program  oversight  process 

5.  New  industry  based  reliability  program  management  standard 

6.  Standard  data  elements  associated  with  the  KPP  and  KSAs 

7.  Reliability  training  for  select  OSD  staff 

8.  NDIA  conference  focusing  on  improving  suitability 

9.  NDIA/DT&E  committee  proposing  changes  to  improve  suitability 
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Agenda 


►  Background 

►  Phase  I  results 

►  Phase  II  efforts 

►  System  Dependent  vs.  System  Independent 

►  Results  and  Conclusions 


Booz  Allen  Hamilton 
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Why  Develop  Service-Wide  Safety  Testing 

Standards? 


►  Moving  forward,  all  weapons/ 
weapon  systems  will  be  developed 
as  joint  systems  vis  a  vis  JCIDS 

►  A  joint  approach  promotes 
consistency  and  will  get  systems 
fielded  sooner.  Reduces  (1)  the 
overall  number  of  tests,  (2)  time  to 
fielding  and  (3)  cost. 


THE  JOINT  STAFF 
WASHINGTON,  D.C.  2Q31B-SOOO 


JROCM  102-05 
20  Hay  2005 


JOINT  REQUIREMENTS 
OVERSIGHT  COUNCIL 


MEMORANDUM  FOR:  Vice  Chief  of  Staff,  US  Army 

Vice  Chief  of  Naval  Operations 

Vice  Chief  of  Staff,  US  Air  Force 

Assistant  Commandant  of  the  Marine  Corps 

Subject:  Safe  Weapons  in  Joint  Warfighting  Environments 


1 .  The  Joint  Requirements  Oversight  Council  (JROC)  approved  the 
establishment  of  a  Joint  Weapons  Safety  Technical  Advisory  Panel  (JWSTAP)  to 
advise  the  Deputy  Director  for  Force  Protection,  J-8,  on  joint  weapons  safety 
issues.  The  JROC  also  approved  the  institution  of  a  Safe  Weapons  in  Joint 
Warfighting  Environments  endorsement  within  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  vetting  process,  upon  the 
development  and  approval  of  a  JWSTAP  charter.  The  Joint  Staff,  J-8, 
Protection  Assessment  Division  will  develop  and  coordinate  the  JWSTAP 
charter  for  joint  approval. 


2.  Because  all  weapons/ weapon  systems  have  the  potential  of  being  deployed 
together  or  employed  in  joint  environments,  weapons  and  weapon  systems  will 
be  considered  joint  systems  within  the  JCIDS  process  unless  they  are  assigned 
the  Joint  Potential  Designator  of  “Independent". 


PETER  PACE 
General,  United  States  Marine  Corps 
Vice  Chairman 
of  the  Joint  Chiefs  of  Staff 


Copy  to: 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
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Phase  I  Results 


►  Collected  over  80  documents  resulting  in  nearly  650  safety  tests 

-  Weapon/explosive  testing  only 

-  MIL-STDs,  STANAGs,  ITOPs,  TOPs,  AECTPs 

-  System-level  and  subsystem/component-level  tests 

-  Service-unique  tests  and  tests  common  to  more  than  one  Service 

-  Many  of  the  tests  identified  are  used  for  other  purposes  other  than  safety 

►  Documents  and  associated  tests  captured  and  categorized  in  a 
Microsoft  Access  database 

-  Service,  type  of  natural  or  induced  environment  that  the  test  simulates,  life 
cycle  phase  that  the  test  simulates,  system/component 

►  Phase  I  clearly  indicates  that  there  are  potential  savings  to  support  a 
Phase  II  effort 
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Phase  I  Results  (cont’d) 


►  Joint  Weapons  Safety  Technical  Advisory  Panel  (JWSTAP),  composed 
of  all  Service  Safety  Board  Reps,  agreed  to  use  the  following  terms  to 
define  a  mode  of  the  weapon 


Mode 


Submode 


►  Wheeled  Land  Vehicles 

►  Rail 

►  Fixed/Rotary  Wing  Aircraft 

►  Operational  Navy  Vessels 

►  Prepo/Merchant  Marine/Commercial 

►  Undersea 

►  Man  Carried 

►  Prior  to  Gov’t  release 

►  After  Gov’t  acceptance 

►  Forklift 

►  Hand  cart 

►  Crane 

►  Man  Carried 

►  Underway  Replenishment  (VERTREP  and  CONREP) 

►  Protected  and/or  Environmentally-Controlled  Land  Based 
Magazines 

►  Unprotected  and/or  Not  Environmentally-Controlled  Land 
Based  Magazines 

►  Tracked  Land  Vehicles 

►  Wheeled  Land  Vehicles 

►  Fixed  Wing  Aircraft 

►  Rotary  Wing  Aircraft 

►  Operational  Navy  Vessels 

►  Undersea 

►  Man  Carried 
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Phase  I  Results  (cont’d) 


Mode  Definitions 

►  Packaging  -  Configuration  of  item  prior  to  transportation.  Includes  test  requirements  prior 
to  government  acceptance  and  repackaging  requirements  for  shipping,  e.g.,  Fleet  Issue 
Unit  Load 

►  Handling*  -  Applies  to  the  use  of  devices,  such  as  forklifts,  hand  carts,  cranes,  underway 
replenishment  and  man  carried,  for  the  breakout,  lifting,  or  repositioning  of  ordnance  or 
explosive  devices  in  order  to  facilitate  storage,  assembly  or  disassembly,  loading  or 
downloading,  or  transporting. 

►  Storage  -  Placing  ordnance  or  explosive  devices  in  temporary  (not  to  exceed  24  hours) 
and  permanent  land  facilities  that  are  either  protected/  environmentally-controlled 
magazines  or  unprotected/open  areas.  This  mode  does  not  include  storage  on  a  ship. 

►  Transportation*  -  The  capability  of  material  to  be  moved  by  wheeled  land  vehicles,  rail, 
fixed/wing  aircraft,  operational  Navy  vessels,  prepo/merchant  marine/commercial, 
undersea,  and  man  carried 


►  Employment*  -  The  strategic,  operational  or  tactical  use  of  weapons  by  tracked/wheeled 
land  vehicles,  fixed/rotary  wing  aircraft,  operational  Navy  vessels,  undersea,  and  man 
carried. 


‘Adapted  from  Joint  Publication  1-02,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms 
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Phase  II  Project  Goals 


►  Draft  JWSTAP  Manual  with  Mode  Checklist  and  Tests  for  each 
Check 

-  Follow  this  manual  to  receive  “safe  weapons  endorsement” 

-  Manual  will  most  likely  refer  to  an  existing  standard.  Deviations  from 
existing  standards  will  be  documented  in  manual. 

-  Manual  will  clarify  any  test  discrepancies  including  passing  criteria 

►  System-independent  tests  only 

►  Need  to  identify  benefit  (e.g.,  cost  savings)  resulting  from  common 
tests 

►  Substantiate  value  to  examine  system  dependent  tests 
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Phase  II  Effort 


►  Refine  Data  Collected 

-  Add  new  database  fields 

-  Review  tests  for  appropriate  categorizations 

►  Identify  Key  Stakeholders 

-  Service  Board  reps 

-  Service-specific  Subject  Matter  Experts  (SMEs) 

►  Conduct  Analysis  on  System-Independent  Tests 

-  Identify  apparent  duplicate,  inconsistent  and  Service-specific  tests 

-  Interview  SMEs  to  validate  initial  findings 


►  Host  stakeholder  workshops  to  obtain  joint  agreement  on  a  standard 
list  of  safety  tests  by  mode 

►  Prepare  final  documentation 
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Phase  II  End  Products 


►  Input  to  CJCSM  31 70 

-  Defines  modes 

-  Provide  draft  language  in  manual  for  ICDs,  CDDs,  CPDs 

►  JWSTAP  Report 

-  Implementing  guidance  to  CJCSM  3170  to  include  specific  tests  per  mode 

-  Rationale  for  implementing  guidance 

-  Contains  details  of  Phase  II  effort 

►  Presentation 

-  Results  of  Study 

-  Summary  of  Report 
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Phase  II  End  Products 


►  JWSTAP  Report 

-  Purpose 

-  Outline 

Executive  Summan 
1.0  Introduction 
2.0  Approach  (Phase  I  and  II) 
3.0  Definitions  of  Modes 
4.0  Tests 

5.0  Analysis  of  Tests 
6.0  Required  Tests  by  Mode 
7.0  Proposed  Language  for 
JCIDS  Documents 
8.0  Summary/Conclusions 

►  Presentation 

-  Results  of  Study 

-  Summary  of  Report 


1.1  Background 

1 .2  Purpose 

1 .3  Scope 

1.4  Assumptions 


4.1  Test  Documentation 

4.2  Test  Classification 

4.3  System  Independent  vs. 
System  Dependent 


5.1  Test  Class  I 

5.2  Test  Class  II 

5.3  Test  Class  XXX 


7.1  CJCSI  3170 

7.2  CJCSM  3170 

7.2.1  Appendix  E  (ICD) 

7.2.2  Appendix  F  (CDD) 

7.3  ICD 

7.4  CDD 
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Phase  II  Report  Outline 


►  Scope 

-  Covers  weapon  and  weapon  container  test  procedures/requirements  in 
all  MIL-STDs,  MIL-SPECs,  STANAGs,  ITOPs,  AOPs 

-  Does  not  cover  any  commercial  standards,  developmental  tests,  system- 
tailored  documents,  IM  tests  or  AECTPs 

-  Analysis  covers  system  independent  tests  defined  by  established  modes 
only 

-  Tests  were  included  as  long  as  the  test  simulated  an  environment  in  one 
of  the  established  modes 

►  Assumptions 

-  Active  standards  are  being  used 

-  All  proposed  required  tests  follow  system-independent  test  documents 

-  All  weapons  being  transported  by  ship  are  in  the  “Transportation”  mode; 
not  the  “Storage”  mode 

-  Assignment  of  test  classifications,  based  upon  test  documentation,  is 
accurate 
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Initial  Capabilities  Document 


The  vision 


Capabilities  Development 
Document 


□  PHS&T  (Transportation) 

□  Wheeled  Land  Vehicles 

□  Rail 

□  Fixed  Wing  Aircraft 

□  Rotary  Wing  Aircraft 

□  Operational  Navy  Vessels 

□  Prepo/Merchant  Marine/Commercial 

□  Undersea 

□  Man  Carried 

□  PHS&T  (Packaging) 

□  Prior  to  Government  Release 

□  After  Government  Acceptance  (Single/Bulk) 

□  PHS&T  (Handling) 

□  Forklift 

□  Handcart 

□  Crane 

□  Man  Carried 

□  Underway  Replenishment  (VERTREP, 
CONREP) 

□  PHS&T  (Storage) 

□  Protected/Environmentally-Controlled  Land 
Based  Magazines 

□  Unprotected/Open  Land  Based  Magazines 

□  Employment 

□  Tracked  Land  Vehicles 

□  Wheeled  Land  Vehicles 

□  Fixed  Wing  Aircraft 

□  Rotary  Wing  Aircraft 

□  Operational  Navy  Vessels 

□  Undersea 

□  Man  Carried 


Drives  System 
_lndependent 
Tests 


System  Independent 
and  Dependent 
Tests 

1.  Joint  Shock  Test 

2.  Joint  Vibration  Test 

3.  Joint  Temperature 
Test 

4.  Joint  EEE  Test 


Drives  System 
Dependent 
Tests 


□  System,  Subsystem, 
All 

□  System-Specific 

□  Ammunition 

□  Cannon 

□  Electric  Initiators 

□  Explosives 

□  Fuze 

□  Power  Sources 

□  Rocket  Motors 

□  Software 

□  Submunitions 

□  Unmanned  Targets 


Capabilities  Production  Document 


Results  of  the  safety  tests 
will  be  detailed  in  the  CPD 


Phase  II  study  is  focusing 
on  identifying  the  system 
independent  tests  for  each 
mode.  Modes  will  be 
defined  in  the  ICD  and 
tests  defined  in  the  ODD. 
Future  work  could  include 
defining  system 
dependent  safety  tests, 
which  would  also  be  used 
to  feed  the  ODD. 
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Required  Tests 


I 


Test  Classifications 


►  High/Low  Temperature 

►  High/Low  Pressure 

►  Contamination  and  Corrosion 

►  Shock  and  Temperature 

►  Temperature  and  Humidity 

►  Temperature/Shock/Humidity 


►  EEE 


►  ESD 

►  Lightning 

►  EMI 

►  HERO 


►  Shock 


►  Short  and  Long  Drops 

►  Vibration 

►  Shock/Vibration 

►  Acoustic 

►  Thermal 

►  Pyro 


Pha^re  II  study  is  focusing 
Identifying  the  system 
Tndependent  tests  for  each 
mode.  Modes  will  be 
defined  in  the  ICD  and 
tests  defined  in  the  CDD. 
Future  work  could  include 
defining  system 
dependent  safety  tests, 
which  would  also  be  used 
to  feed  the  CDD. 


Every  System  going  through 
development  will  be  required  to  do 
standard  joint  tests.  There  will  be  a 
test  for  each  test  classification. 
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System  Independent  vs.  System  Dependent 


►  System  Independent 

-  Certain  tests  you  do  regardless  of  what  is  in  the  weapon  system. 

-  Example:  Shock,  Drop,  Vibration. 

►  System  Dependent 

-  Tests  that  are  driven  by  specific  components  of  a  system. 

-  Tests  are  more  clearly  defined  once  the  system  is  mature. 

-  Example:  Fuze,  Ammunition,  Explosives. 


Our  findings  show  that  Services  use  system-dependent 
tests  for  standard  system  independent  tests. 


Booz  Allen  Hamilton 
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System  Independent  Documents 


Doc  Type 

TD 

Number 

TD  Name 

49  CFR 

178.6 

Testing  of  Non-bulk  Packagings  and  Packages 

HOP 

1-2-511 

Intersystem  Electromagnetic  Compatibility  Requirements,  System  Testing 

HOP 

4-2-601 

Drop  Tests  for  Munitions 

HOP 

4-2-828 

Ballistic  Shock  Testing 

MIL-STD 

461 

Requirements  for  the  Control  of  Electromagnetic  Interference  Characteristics 
of  Subsystems  and  Equipment 

MIL-STD 

464 

EEE  Requirements  for  Systems 

MIL-STD 

648 

Design  Criteria  for  Specialized  Shipping  Containers 

MIL-STD 

810 

Environmental  Engineering  Considerations 

MIL-S 

901 

Shock  Tests,  High  Impact  Shipboard  Machinery,  Equipment,  and  Systems 
Requirements 

STANAG 

4239 

Electrostatic  Discharge  Munitions  Test  Procedures 

STANAG 

4327 

Lightning,  Munition  Assessment  and  Test  Procedures 

STANAG 

4375 

Safety  Drop,  Munition  Test  Procedure 

We  have  reduced  the  number  of  documents  from  86  to  12. 
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System  Independent  Tests  by  Test  Classification 


►  Contamination  &  Corrosion  -  6 

►  EEE-40 

►  Icing  - 1 

►  Impact  -  5 

►  Leak  (internal)  -  5 

►  Lifting  - 1 

►  Pressure  -  High  -  2 

►  Pressure  -  Low  - 1 

►  Shock  -  4 

►  Shock  -  Acoustic  -  2 

►  Shock  -  Mechanical  -  3 


►  Shock  -  Pyro  - 1 

►  Shock  and  Temperature  -  2 

►  Shock/Vibration  - 1 

►  Shock-Mechanical  (long  drop)  -  6 

►  Shock-Mechanical  (short  drop)  -  9 

►  Temperature  -  High  -  3 

►  Temperature  -  Low  - 1 

►  Temperature  and  Humidity  - 1 

►  Temperature  Shock  Humidity  - 1 

►  Tiedown  -  2 

►  Vibration  - 10 


We  have  reduced  the  number  of  tests  for  analysis  from  650  to  107. 
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Preliminary  Findings 


►  JWSTAP  members  have  little  involvement  with  or  knowledge 
of  the  test  community 

►  Many  active  standards  are  not  being  used 

-  No  one  that  we  have  talked  with  are  using  AECTPs 

-  Ratified  STANAGs  are  not  being  used  by  US  (e.g.,  STANAG  4239, 
Electrostatic  Discharge  Munitions  Test  Procedures) 

-  There  is  a  tendency  not  to  use  STANAGs  at  all 

►  System  dependent  tests  are  used  for  system  independent 
tests 

-  There  is  no  system  independent  short  drop  tests 

-  MIL-STD-464  ESD  test  refers  to  MIL-STD-331,  a  fuze  standard 
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Preliminary  Findings  (cont’d) 


►  ESD  Testing 

-  When  testing  Helicopter  Borne  ESD,  the  Navy  and  Army  have 
different  test  procedures.  The  Navy  tests  bare  systems  whereas  the 
Army  tests  systems  in  a  packaged  configuration. 

-  STANAG  4239,  Electrostatic  Discharge  Munitions  Test  Procedures,  is 
not  the  primary  standard  being  used  by  the  Services. 

-  The  Army  uses  inert  EEDs  for  testing  whereas  the  Navy  uses  live 
EEDs  for  testing.  The  Navy  will  not  accept  the  Army’s  test  results  in 
these  cases. 

►  Other  EEE  Testing 

-  MIL-STD-461  tests  are  being  used  at  the  system/platform  level  rather 
than  MIL-STD-464.  Root  issue  is  a  standardize  definition  of  “system.” 
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Preliminary  Findings  (cont’d) 


►  Vibration 

-  Some  new  programs  develop  Project  Peculiar  Documents  (PPDs). 
The  PPDs  establish  vibration  requirements  different  from  any  MIL- 
STD  or  STAN  AG. 

►  Mechanical  Drop 

-  There  is  no  standard  short  test  being  used  by  the  Services. 
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Conclusions 


►  Defined  approach  and  methodology  to  conduct  analysis. 

►  Currently  interviewing  Stakeholders  and  SMEs  to  validate  test  data  and  test 
classifications. 

►  Preparing  for  facilitated  meetings  with  Stakeholders  and  SMEs  to  gain 
consensus  on  standard  joint  tests. 

►  On  track  to  prepare  manual  and  outline  of  final  report  to  be  presented  to  the 
JWSTAP  in  Nov  07. 

►  Phase  II  project  is  completed  by  Feb  2008. 
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BACK-UP 
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System  Independent  Tests  by  Test  Classification 
Contamination  &  Corrosion 


p 

H 

s 

T 

E 

Test  Name 

Test  Number 

Doc  Type 

TD  Number 

X 

X 

X 

X 

X 

Acidic  Atmosphere 

Method  518 

MIL-STD 

810 

X 

X 

X 

X 

X 

Contamination  by  Fluids 

Method  504 

MIL-STD 

810 

X 

Fungus 

Method  508.5 

MIL-STD 

810 

X 

X 

X 

Salt  Fog 

Method  509.4 

MIL-STD 

810 

X 

X 

X 

X 

X 

Salt  Fog  Test 

5.4.1 

MIL-STD 

648 

x-u 

Sand  and  Dust 

Method  510.4 

MIL-STD 

810 

Booz  Allen  Hamilton 


22 


System  Independent  Tests  by  Test  Classification 


p 

H 

s 

T 

E 

Test  Name 

Test 

Number 

Doc  Type 

TD 

Number 

X 

Conducted  Emissions,  Antenna  Terminal  lOkhz  to  40ghz 

CE106 

MIL-STD 

461 

X 

Conducted  Emissions,  Power  Leads  lOkhz  to  lOmhz 

CE102 

MIL-STD 

461 

X 

Conducted  Emissions,  Power  Leads  30hz  to  lOkhz 

CE101 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Antenna  Port  Rejection  of  Undesired  Signals 

30hz  to  20ghz 

CS104 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Antenna  Port,  Cross  Modulation  30hz  to  20ghz 

CS105 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Antenna  Port,  Intermodulation  15khzto  lOghz 

CS103 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Bulk  Cable  Injection  lOkhz  to  200mhz 

CS114 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Bulk  Cable  Injection,  Impulse  Excitation 

CS115 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Damped  Sinusoidal  Trans  Cables  and  Power 

Leads,  10  kHz  to  100  MHz. 

CS116 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Power  Leads  30hz  to  150  kHz 

CS101 

MIL-STD 

461 

X 

Conducted  Susceptibility,  Structure  Current 

CS109 

MIL-STD 

461 

X 

Electrical  Bonding 

5.1 

MIL-STD 

464 

X 

Electromagnetic  Pulse  (EMP) 

5.5 

MIL-STD 

464 

X 

X 

X 

X 

X 

Electromagnetic  Radiation  Hazards  (EMRADHAZ) 

5.8 

MIL-STD 

464 

X- 

MC 

X- 

RWA 

X- 

MC&RWA 

Electrostatic  Discharge  (Personnel  and  Helicopter) 

A-1 

STANAG 

4239 
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System  Independent  Tests  by  Test  Classification 

EEE  (cont’d) 


p 

H 

s 

T 

E 

Test  Name 

Test 

Number 

Doc 

Type 

TD 

Number 

X 

X 

X 

X 

X 

Emission  Control  (EMCON) 

5.13 

MIL-STD 

464 

X 

X 

X 

X 

X 

EMRH 

2.1 

TOP 

1-2-511 

X 

X 

X 

X 

X 

EMRO 

2.2 

TOP 

1-2-511 

X 

X 

X 

X 

Evaluation  of  the  Hazards  Caused  By  Shock  Excitation  and  Ground  Voltage  Transients 

B-2-1 

STANAG 

4327 

X 

X 

X 

X 

Explosives  and  Fuel  Hazards  Assessment 

B-3-1 

STANAG 

4327 

X 

X 

X 

X 

X 

External  Ground 

5.11 

MIL-STD 

464 

X 

External  RF  EME 

5.3 

MIL-STD 

464 

X 

X 

X 

X 

Group  IV  Test  Methods 

C-7-1 

STANAG 

4327 

X 

Intra-System  Electromagnetic  Compatibility  (EMC) 

5.2 

MIL-STD 

464 

X 

X 

X 

X 

Lightning  Test  Method 

C-1 

STANAG 

4327 

X 

X 

X 

X 

Lightning  Test  Waveforms  for  use  with  the  Test  Methods  Given  in  Appendices  C1-C7 

B-4-1 

STANAG 

4327 

X 

X 

X 

X 

Methods  for  Detecting  Sparking 

C-8-1 

STANAG 

4327 

X 

Radiated  Emissions  Magnetic  Field 

RE101 

MIL-STD 

461 

X 

Radiated  Emissions,  Antenna  Spurious  &  Harmonic 

RE103 

MIL-STD 

461 

X 

Radiated  Emissions,  Electric  Field 

RE102 

MIL-STD 

461 

X 

Radiated  Susceptibility,  Electric  Field 

RSI  03 

MIL-STD 

461 

X 

Radiated  Susceptibility,  Magnetic  Field 

RS101 

MIL-STD 

461 

X 

Radiated  Susceptibility,  Transient  Electromagnetic 

RSI  05 

MIL-STD 

461 
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System  Independent  Tests  by  Test  Classification 

EEE  (cont’d) 


p 

H 

s 

T 

E 

Test  Name 

Test 

Number 

Doc 

Type 

TD 

Number 

X 

X 

X 

X 

Requirements  for  Energetic  Material  Hazard  Assessment  Tests-Solid  Explosives 

C-4-1 

STANAG 

4327 

X 

X 

X 

X 

Requirements  for  Equipment 

C-3-1 

STANAG 

4327 

X 

X 

X 

X 

Requirements  for  Group  II  Tests  on  Parts  of  Weapon 

C-2-1 

STANAG 

4327 

X 

X 

X 

X 

Requirements  for  Group  II  Tests  on  Whole  Weapons 

C-1-1 

STANAG 

4327 

X 

X 

X 

X 

X 

Static  Electricity  Hazard 

2.1 

TOP 

1-2-511 

X 

X 

X 

X 

X 

Static  Electricity  Operational 

2.2 

TOP 

1-2-511 

X 

Subsystems  and  Equipment  Electromagnetic  Interference  (EMI) 

5.6 

MIL-STD 

464 
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System  Independent  Tests  by  Test  Classification 
Icing,  Impact  Leak  (internal),  Lifting 


p 

H 

s 

T 

E 

Test 

Classification 

Test  Name 

Test  Number 

Doc 

Type 

TD 

Number 

X 

Icing 

Icing/Freezing  Rain 

Method  521.2 

MIL-STD 

810 

X-Rail 

Impact 

Impact  Test  (stacked) 

5.2. 7.1 

MIL-STD 

648 

X 

Impact 

Incline-Impact 

Appendix  L 

MIL-STD 

648 

X 

Impact 

Pendulum  Impact  Test 

5.2.7 

MIL-STD 

648 

X 

X 

Impact 

Rollover  Test 

Appendix  K 

MIL-STD 

648 

X 

Impact 

Stacking  Test 

178.606 

49  CFR 

178.6 

X 

Leak  (internal) 

Explosive  Atmosphere 

Method  511.4 

MIL-STD 

810 

X 

X 

Leak  (internal) 

Immersion 

Method  512.4 

MIL-STD 

810 

X 

Leak  (internal) 

Leak  Proofness  Test 

178.604 

49  CFR 

178.6 

X 

X 

Leak  (internal) 

Leak  Test 

5.6.2 

MIL-STD 

648 

X 

X 

X 

X 

X 

Leak  (internal) 

Rain 

Method  506.4 

MIL-STD 

810 

X- 

Forklift 

Lifting 

Forklift  Truck  (Fully  Captive  Fork  Tine 
Enclosures)  Compatibility  Test 

5.9 

MIL-STD 

648 

Booz  Allen  Hamilton 
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System  Independent  Tests  by  Test  Classification 
Pressure,  Shock,  Shock  and  Temp,  Shock/Vibration 


p 

H 

s 

T 

E 

Test  Classification 

Test  Name 

Test  Number 

Doc  Type 

TD 

Number 

X 

Pressure  -  High 

Hydrostatic  Pressure  Test 

178.605 

49  CFR 

178.6 

X 

Pressure  -  High 

Pressure  test 

5.5.2 

MIL-STD 

648 

X 

X 

X 

Pressure  -  Low 

Low  Pressure  (Altitude) 

Method  500.4 

MIL-STD 

810 

X-ONV 

X 

Shock 

Heavy  Weight  Shock 

3.1.2c 

MIL-S 

901 

X-ONV 

X 

Shock 

Light  Weight  Shock 

3.1.2a 

MIL-S 

901 

X-ONV 

X 

Shock 

Medium  Weight  Shock 

3.1.2b 

MIL-S 

901 

X 

X 

Shock 

Shipboard  Shock  Test 

5.2.9 

MIL-STD 

648 

X 

Shock  -  Acoustic 

Acoustic  Noise 

Method  515.5 

MIL-STD 

810 

X 

Shock  -  Acoustic 

Ballistic  Shock 

4 

ITOP 

4-2-828 

X 

X 

Shock  -  Mechanical 

Acceleration 

Method  513.5 

MIL-STD 

810 

X 

X 

Shock  -  Mechanical 

Ballistic  Shock 

Method  522 

MIL-STD 

810 

X 

X 

X 

Shock  -  Mechanical 

Shock 

Method  516.5 

MIL-STD 

810 

X 

X 

Shock  -  Pyro 

Pyroshock 

Method  517 

MIL-STD 

810 

X 

X 

X 

X 

X 

Shock  and  Temperature 

Temperature  Shock 

Method  503.4 

MIL-STD 

810 

X 

Shock  and  Temperature 

Vibro-Acoustic/Temperature 

Method  523.2 

MIL-STD 

810 

X 

Shock/Vibration 

Gunfire  Vibration 

Method  519.5 

MIL-STD 

810 

Booz  Allen  Hamilton 
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System  Independent  Tests  by  Test  Classification 
Shock-Mechanical  (long  and  short  drops) 


p 

H 

s 

T 

E 

Test  Classification 

Test  Name 

Test 

Number 

Doc 

Type 

TD 

Number 

X 

Shock-Mechanical  (long  drop) 

Drop 

8a 

STANAG 

4375 

X 

Shock-Mechanical  (long  drop) 

Safety  drop  test 

5.2.10 

MIL-STD 

648 

X 

Shock-Mechanical  (long  drop) 

Simulated  High  Velocity  Parachute  Drop 

4.4 

ITOP 

4-2-601 

X 

Shock-Mechanical  (long  drop) 

Simulated  Low  Velocity  Parachute  Drop 

4.3 

ITOP 

4-2-601 

X 

Shock-Mechanical  (long  drop) 

Twelve  Meter  Drop 

4.1 

ITOP 

4-2-601 

X 

Shock-Mechanical  (short  drop) 

Cornerwise-drop  (rotational)  test  (12-15  in 
drop) 

5.2.4 

MIL-STD 

648 

X 

Shock-Mechanical  (short  drop) 

Drop  Test 

178.603 

49  CFR 

178.6 

X 

X 

X 

X 

X 

Shock-Mechanical  (short  drop) 

Drop  Test  (free  fall) 

5.2.3 

MIL-STD 

648 

X 

X 

X 

X 

X 

Shock-Mechanical  (short  drop) 

Edgewise-drop  (rotational)  Test 

5.2.5 

MIL-STD 

648 

X 

Shock-Mechanical  (short  drop) 

Mechanical  Handling  Test 

Appendix 

P 

MIL-STD 

648 

X 

X 

Shock-Mechanical  (short  drop) 

Shock  Test 

5.10.3 

MIL-STD 

648 

X 

Shock-Mechanical  (short  drop) 

Three  Meter  Drop 

4.2 

ITOP 

4-2-601 

X 

Shock-Mechanical  (short  drop) 

Tipover  test 

5.2.6 

MIL-STD 

648 

X-UR 

Shock-Mechanical  (short  drop) 

Transfer-at-sea  Shock  Test 

5.2.8 

MIL-STD 

648 

Booz  Allen  Hamilton 
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System  Independent  Tests  by  Test  Classification 
Temp,  Temp  and  Humidity,  Temp-Shock-Humidity,  Tiedown,  Vibration 


p 

H 

s 

T 

E 

Test  Classification 

Test  Name 

Test 

Number 

Doc 

Type 

TD 

Number 

X 

Temperature  -  High 

Fire  test,  external  source. 

5.11 

MIL-STD 

648 

X 

X 

X 

X 

X 

Temperature  -  High 

High  Temperature 

Method  501.4 

MIL-STD 

810 

x-u 

Temperature  -  High 

Solar  Radiation  (Sunshine) 

Method  505.4 

MIL-STD 

810 

X 

X 

Temperature  -  Low 

Low  Temperature 

Method  502.4 

MIL-STD 

810 

X 

X 

Temperature  and  Humidity 

Humidity 

Method  507.4 

MIL-STD 

810 

X-A 

Temperature  Shock  Humidity 

Temperature,  Humidity,  Vibration  and  Altitude 

Method  520.2 

MIL-STD 

810 

X-Road 

Tiedown 

Hoisting  Fitting  and  Tiedown  Attachment  Points 

5.8 

MIL-STD 

648 

X-Road 

Tiedown 

Tiedown  Strength  Test 

5.8.4 

MIL-STD 

648 

X 

X 

Vibration 

Endurance 

5.2. 1.4.6 

MIL-STD 

167-1 

X 

X 

Vibration 

Endurance  Test  for  Mast  Mounted  Equipment 

5.1 .2.4.7 

MIL-STD 

167-1 

X 

X 

Vibration 

Exploratory  Vibration 

5.1. 2.4.2 

MIL-STD 

167-1 

X 

X 

X 

X 

X 

Vibration 

Random  Vibration 

5.3.4 

MIL-STD 

648 

X 

Vibration 

Repetitive  Shock  Test 

5.2.2 

MIL-STD 

648 

X 

Vibration 

Repetitive  Shock  Test  (stacked) 

5.2.2. 1 

MIL-STD 

648 

X 

Vibration 

Variable  Frequency 

5.1 .2.4.3 

MIL-STD 

167-1 

X 

Vibration 

Vibration 

178.608 

49  CFR 

178.6 

X 

X 

Vibration 

Vibration 

5.3 

MIL-STD 

648 

X 

X 

X 

X 

X 

Vibration 

Vibration 

Method  514.5 

MIL-STD 

810 

Booz  Allen  Hamilton 
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Systems  and  Software  Engineering 

Overview 

October  23,  2007 


Version  0.9 


Mark  D.  Schaeffer 

Director,  Systems  and  Software  Engineering 
Office  of  the  Deputy  Under  Secretary  of  Defense  (A&T) 
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SE  Challenges  ...circa  2003 


>  Lack  of  Uniform  SE  Understanding  at  Department  level 

•  Policy,  implementation,  incentives,  product/process 
balance,  life-cycle  focus 

>  Lack  of  Uniform  Understanding  of  SE  in  the  Community- 
at-Large 

•  Scope  of  SE,  understanding  of  how  to  implement  SE, 
what  makes  a  good  systems  engineer,  alignment  of 
management  and  SE  processes,  multiple  standards 
and  models,  alignment  of  practitioner  communities 

>  Increasing  system  complexity 

•  Integrated  systems  ...  higher  levels  of  integration 

>  SE  Resources  -  “We  have  a  pipeline  problem” 


Our  Challenge:  Execute  the  “Big  Picture ” 


2 


DoD  Systems  Engineering  Response 


>  USD(AT&L)  Imperatives 

•  Provide  a  context  for  decisions  on  individual 
programs 

•  Achieve  credibility  and  effectiveness  in  acquisition 
and  logistics  support  processes 

•  Help  drive  good  SE  practices  back  in  how  we  do 
business 

>  Organizational  response:  new  SE  organization 

•  Institutionalize  SE  across  the  Department 

•  Set  implementation  policy,  capture  best  practices, 
standards  for  Education  and  Training 

•  Emphasis  on  system  assessment  and  support 
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What  We  Have  Done  To  Revitalize 
Systems  Engineering  2003  -  Today 

>  Issued  DoD-wide  SE  policy  -  focused  effort  on  up  front,  sound 
technical  planning;  issued  safety  policy 

>  Issued  guidance  on  Systems  Engineering,  test  and  evaluation  (T&E), 
software,  and  safety 

>  Worked  with  Defense  Acquisition  University  to  revise  Systems  Engineering 

curricula  --  currently  revising  T&E  and  enabling  career  fields  curricula 

and  Software 

>  Established  Systems  EngineeringTorum — senior-level  focus  within  DoD 

>  Integrated  development  testing,  software/system  assurance,  system  of 
systems,  and  open  systems  into  revitalization  efforts 

>  Instituting  a  renewed  emphasis  on  modeling  and  simulation 

>  Leveraging  closer  working  relationships  with  industry  and  academia 

>  Instituting  system-level  assessments  in  support  of  OSD  major  acquisition 
program  oversight  role 
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System  Engineering  Revitalization  Effort 


*  New/Updated  Courses 

Career  Field  Training 

Fundamentals  of  Systems  Engineering  Course 
Systems  Planning,  Research,  Development  and  Engineering 
Course  (Intermediate  A&B  and  Advanced) 

Test  &  Evaluation  Courses 
Course  updates  to  address  Safety 
Continuous  Learning  Courses 
System  Safety 
Modeling  and  Simulation 
Technical  Planning 
Reliability  and  Maintainability 
Technical  Reviews 
Modular  Open  Systems 
Trade  Studies 


2003 


2004 


2005 


2006 


2007 


2008 


Current  State  of  DoD  Systems  Engineering 


>  We  have  revitalized  Systems  Engineering  Policy,  Guidance, 
Education  and  Training... 

>  We  have  driven  good  systems  engineering  practices  back  into  the 
way  the  acquisition  community  does  business,  and  have  had  a 
positive  impact  on  programs... 

>  We  have  expanded  the  boundaries  to  include  increasingly  important 
enablers  for  sound  SE  application... 

>  We  have  a  rigorous  process  to  capture  what  went  wrong... 

>  ...but  failed  to  change,  root  cause  behavior  that  leads  to  programs 
that  do  not  meet  cost,  schedule,  and  performance 
expectations... adequate  maturity  at  program  initiation 


Much  Accomplished  -  Much  to  Do! 
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Mr.  Carl  R.  Siel,  Jr. 
ASN(RDA)  Chief  Systems  Engineer 
carl.siel@navy.mil 


Topics 


Current  Challenges 
Progress 

Emergent  Challenges 


Department  of  Navy  Challenges 


Eda 

Chief 

Systems 

Engineer 


♦  Reverse  erosion  of  domain  knowledge  within  DoN 

♦  Increase  our  knowledge  of  the  shipbuilding  process 

-  “...understand  how  to  integrate  design  and  production  technology 
into  an  acquisition  process  that  industry  can  execute.” 

♦  Establish  a  deep  knowledge  of  systems  engineering  and  a 
profound  understanding  of  acquisition  process 

-  “Systems  engineering  is  key  to  ensuring  that  each  ship  is 
configured  to  optimize  the  fleet. 

-  The  Navy  does  not  fight  a  ship  by  itself.  It  wages  war  as  part  of  an 
Expeditionary  Strike  Group  or  a  Carrier  Strike  Group. 

-  And  those  strike  group  formations  are  part  of  even  larger  Joint 
operations. 

-  All  this  implies  a  need  for  integration  of  elements  and  capabilities.  ” 


(Adapted  from  SECNAV  speech  to  the  Sea  Air  Space  Exposition  on  3  April  2007) 


RDA  CHSENG  Off-Site  Brief  (3-4  Apr  07) 


3 


DoN  Systems  Engineering  Hierarchy  f^£MS 


Mission 


Force  Focused 


\ 


I 


Systems  and 
Components 

Design,  Build,  Test  Focused 


Translates 

Operational  Concepts 
Capabilities 


System  of  Systems 

Net  Centric  Integration 
Platform  Integration 

Capability  Focused 


Translates 

Capabilities  System 
Requirements 


Translates 
System  Requirements 
End  Items 


:ER 


V^e 


Mission 


rare 


Platform  +  Net  Centric 


Electronic  Warfare 


>  SoS"^ 
integration 

Platform  ^  Net  Centrii 


Air  Land  Sea 

...  A 


IC4I,  IWS, ... 


Systems,  Components, 
Equipment,  Materials, 
Software,  etc. 


Must  consider  the  Hierarchy  and  DOTMLPF  over  Time 


RDA  CHSENG  Off-Site  Brief  (3-4  Apr  07) 
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Steps  to  Address  Challenges  SEF 

1  Bystems 

_ _ _j  Engineer 

♦  Navy  must  re-assert  its  control  over  the  entire  acquisition 
process 

-  “Control  over  the  process  means  that  the  Navy  must  make  the 
selections  of  key  trade-offs  -  performance,  crew  size,  logistics 
support,  cost,  and  schedule. 

-  Added  to  that  consideration  is  the  fact  that  ships  do  not  operate  in 
isolation  -  -  they  operate  with  shore  and  air  components. 

-  These  other  factors  are  highly  relevant,  so  it  is  very  important  that 
the  Navy  take  all  factors  into  consideration  and  exercise  control 
over  the  decision  process.” 

♦  The  Navy  must  define  the  design  constraints  to  optimize 
the  overall  capability  of  the  Fleet 

-  “...it  is  the  Navy’s  responsibility  to  optimize  the  fleet’s  capabilities. 

-  Such  optimization  might  include  common  standards;  preferred 
components  and  subsystems;  mission  modularity;  and  open 
architecture." 

(Adapted  from  SECNAV  speech  to  the  Sea  Air  Space  Exposition  on  3  April  2007) 


RDA  CHSENG  Off-Site  Brief  (3-4  Apr  07) 
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Progress 


♦  People,  Process,  Tools,  Standards 

-  Exercising  Mission  Level  ■  Systems  of  Systems  Engineering 

•  Mine  Warfare  and  Anti  Submarine  Warfare  Missions 

-  Systems  Engineering  Technical  Review  Process 

•  Consolidating  into  a  comprehensive  process 

•  Alignment  of  System  Engineering  Plans 

•  Increased  management  of  technical  standards 

•  Technical  Authority  and  Competency  Alignment 

-  Systems  Engineering  Competencies 

•  Personnel  Knowledge,  Skills,  and  Abilities  (KSA) 

•  Education,  Training,  and  Experience 

•  Air,  Sea,  Land,  and  Net-Centric  Mission  Systems 


L 'Rda 

Chief 

Systems 

Engineer 


♦  PR  09  Systems  Engineering  Revitalization 

-  $  150  M  increase  over  FY  09  - 13 

•  Enhance  People,  Process,  Tools,  and  Standards 


RDA  CHSENG  Off-Site  Brief  (3-4  Apr  07) 
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Emergent  Challenges 


Eda 

Chief 

Systems 

Engineer 


♦  Securing  and  Assuring  the  “System” 

-  Protecting  Program  and  Operational  Information 

-  Maintaining  confidence  in  our  Products 

•  Network  and  Software  Vulnerabilities 

•  Information  Security  and  Assurance 

•  Anti-Tamper  -  Re  Engineering 

-  Safe  and  Assured  Operations 

•  Weapon  Safety 

•  Air  Worthiness  and  Safety  of  Flight 

•  Submarine  Safety 

•  Surface  Ship  Certification 

•  Information  Security  and  Assurance 

-  Prevent  the  Loss  of  Life  and  Property 

Consumers  -  Suppliers  -  Users  are  part  of  the  Equation 


RDA  CHSENG  Off-Site  Brief  (3-4  Apr  07) 


SE  View  from  Army 

23  October  2007 


Douglas  K.  Wiltsie 

Assistant  Deputy 

Acquisition  and  Systems  Management 

Office  of  the  Assistant  Secretary  of  the  Army 
Acquisition  Logistics  and  Technology 


Sec.  Bolton’s  Challenges 


•  Systems  Engineering: 

-  Does  not  help  us  politically 

-  Does  not  stabilize  funding 

-  Does  not  belong  in  the  Requirements  Process 

-  Does  not  clearly  address  System  of  Systems 


Army  System  Engineering  Policy 


The  Army  System 
Engineering  program 
and  policy  approved 
(13  June  2005) 


•  Requires  a  SEP  for  each  program 

•  Establishes  a  System  Engineer  within 
each  program  and  PEO 

•  Establishes  Army  System  Engineering 
Forum  (ASEF) 

•  Establishes  peer  review  at  all  major 
technical  reviews 

•  Establishes  the  PEO  as  the  SEP 
approval  authority 


DnwmwT  chf  the  ahht 
tyr  rm  Gf  it*  uw 
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Current  Focus 


System  Engineering  is  being  done  in  Army  programs;  we 
need  to  ensure  that  it  is  consistent  and  consistently 
followed  across  the  PEOs 

Training  is  widely  available  but  standards  need  to  be 
established;  we  need  to  identify  what’s  available  and  tailor 
to  PEO/PM  needs 

Requirements  are  done  outside  of  the  SE  process;  engage 
TRADOC  on  C4ISR  and  BC  migration  and  identify  new 
processes  for  SoS  development 

Integrate  Science  and  Technology  into  Systems 
Engineering  revitalization 

Investigate  establishment  of  a  SoS  Eng  and  Architecture 
Organization 


Capability  Based  Acquisition 


Army  is  transitioning  to  more  and  more 
Capability  Based  acquisition. 


•  Software  blocking  -  Ensures  end  to  end  operability  for  all  current  and  future  battle  command 

•  Future  Combat  System-  1st  Army  System  of  Systems  capability  based  acquisition  focused  on  developing  and  procuring  a  brigade  level  set  of  equipment 

•  Army  Missile  and  Space  -  Develops  the  requirements  and  products  to  provide  Air  and  Missile  Defense  capability 

•  Joint  Network  Node  (JNN)  to  Warfighter  Information  Network-Tactical  (WIN-T)  -  Current  AOR  network  interoperability  with  future  network. 

•  Counter  rocket  and  mortar  -  continual  evolution  of  requirements 

•  Counter  Improvised  Explosive  Devices  -  evolving/changing  requirements  and 
environments. 

•  Force  Protection  -  continual  evolution  of  requirements 


Battle  Command  -  transition  from  current  to  future  battle  command  capabilities 


Introduction:  The  Paradigm  Shift 


Well  Bounded  System 


‘MEGA  SYSTEM” 


| concept  >A 

|DevelopmentA>  /\ 

|Productiot^> 


Must  Change 

Perspective 
Boundaries 
Process 
People  (KSAs) 
Tools 


Support 


> 


Year  1 


Spill  Oil  1*3, 

Curren 

Force" 


Year  8 

Transform  to  provide  multiple  innovative 
overmatching  capability  options  to  the  customer 
Evolve  supporting  processes  into  an  integrated,  DP  &T 

cross  commodity,  cross  community  SOS 
environment. 


Delivery  of  Right  Capabilities  on  Schedule  on  Budget 


Requirements  Generation 


Goal:  To  Integrate  SE  into  the  Requirements  Development 

Process,  Especially  for  Complex  Interdependent  Programs 


•  Establish  methods  to  support  requirements  generation  at  the  System  of 
Systems  or  Enterprise  Level  and  help  define  the  trade  space 

-  ASA(ALT)/TRADOC  Capability  Engineering  Framework  (CEF) 
Initiative  for  engineering  the  requirements/acquisition  interface 

-  Program  Execution  Working  Group  for  cross  PEO/TRADOC  SE  for 
C4ISR  migration 

-  Software  Blocking 

-  Ground  Soldier  System  minimum  essential  capability 

•  Stepping  stone  to  Joint  System  of  Systems  requirements 

-  Generation  Process  (e.g.  SIAP,  SIGP,  JBMC2,  NCOE) 

-  Without  Joint  Level  overarching  requirements,  System  Level 
requirements  could  be  met  and  still  not  meet  Joint  Requirements 


Army  Strategic  S/W  Improvement  Program 


Goal:  To  dramatically  improve  the  acquisition 
of  software  intensive  systems 

Objectives: 

-Foster  migration  to  a  model-based  system  and  software 
acquisition  process  improvement 

-Institutionalize  broad-based  oversight,  management,  and 
technical  expertise 

-Apply  an  integrated  system  and  software  engineering 
approach  to  programs  and  improvement 

-Systematically  incorporate  lessons  learned,  best  practices, 
and  new  technology  into  policies,  practices  and  processes 


